Hello, what would be the best way to override and add a custom

TaskDataStore

with metaflow extensions?

You can add it there: https://github.com/Netflix/metaflow-extensions-template/blob/master/metaflow_extensions/org/plugins/mfextinit_org.py#L34

Please correct me if I’m wrong here but I think that’s a place if you wanted to create custom classes that extend

DataStoreStorage

. I’d like to create a custom TaskDataStore.

ah you are correct. Sorry I misread. What is your use case for a custom TaskDataStore?

I’d like to be able to cleanly hook into the

save_artifacts

and

load_artifacts

to serialize objects with something other than

pickle

this is very apropos. We have been talking about this internally.

yes that is correct. effective and custom when needed

Today the best way to store a dataframe without hacking any internals is to write a function, say

to_parquet_bytes

,that writes your parquet file and loads the bytes in memory, you can then store the bytes as an artifact and use

from_parquet_bytes

in another step.

yea we’ve been doing that, changing the dataframe object into parquet bytes and then pickling it, but I’d like to access some of these files directly from s3 paths as well

so rn, we don’t have support for that directly (to override the taskdatastore). I will think if there is a very hacky way of doing it but defintiely won’t even be “somewhat supported”. The idea of having custom serializers/deserializers is a good idea and I think we can work on this but it’s not there today.

its a little cumbersome to unpickle the file and then convert the parquet bytes into the dataframe

We've been discussing a bit about how we can support some custom serialization. Here at Netflix we have a simple way to read and write tables from a dataframe, so tables are usually the backing storage for a dataframe, rather than a data artifact.

sorry, could you elaborate a little on what you mean by tables here? did you create a new pluggin?

Sure, so we have a managed iceberg data warehouse for tabular data. We've written some tools in our netflix metaflow extensions to read and write data to the warehouse. So that ends up being a convenient place to store any tabular data. You can see a talk about it here -