Hello, I'm trying to use the client API to access data artifacts produced from my runs using the documentation here. Specifically I'm using a command like this to get the data:

test_data = step.task.data.raw_data["data"]

Does anyone know where this data is downloaded on disk? Can I configure it to download the data at a specific location? Currently, it downloads it to a partition which has limited memory resulting in OOM errors

It uses pickle to save/load data behind the hood, therefore it loads it to memory. Unless the data is saved incrementally it can’t be loaded incrementally to overcome the OOM errors. Instead I would recommend that you store a pointer to the data as a data artifact and store the actual data in another format that enables you to load it in batches. Or you could also just partition the data yourself and then store pointers to all batches as data artifacts then load them incrementally.

Does the pickling consume disk space as well?

@fresh-wall-61998 you can set

METAFLOW_CLIENT_CACHE_PATH

to a different directory than

/tmp