Hi Everyone, I'm new to Metaflow and have a quick question. If in one of my steps I call a function that produces an output file that might be very large how should I properly handle passing this to the next steps and having it cached properly? I could just load the file into memory and store it as an artifact, but the file could be larger than memory. Alternatively, I could just save the file path as an artifact, but then it seems the caching property wouldn't work properly as the file may change but the file path may stay the same. Any advice for how to address this? Is it possible to write an artifact in pieces perhaps? Or to set the hash value of an artifact manually?

Hi Julian 🙂 If you are using an object store (S3, gcs, azure blob, minio) not on the memory-limited device where you run flows, you could put the artifact there at the end of the producing task. This can work well orthogonal to your idea of breaking the artifact into pieces; however there is still going to be a lot of expensive serialization happening if you

self.

all those pieces; instead you may want to chunk the data, then put the chunks in a cloud storage bucket or on your local disk like you suggest.

Thanks for getting back to me! I think I'm still a bit confused on what to do for files that don't fit in the machined memory (i.e. RAM). Here is an example:

class SomeFlow(FlowSpec):
    ...

    @step
    def some_step(self):
        output_file_path = ...
        function_that_produces_file(output_file_path)

        with open(output_file_path, "r") as f:
            self.file = f.read()

this works if the file stored in

output_file_path

is small enough to be read into RAM, but what should I do when it does not fit into RAM? I think chunking might help, but not really sure how to do that.

Gotcha: a few questions to get you a more detailed suggestion 1. Can you share a bit about what

function_that_produces_file

does? 2. What are you trying to do with the object loaded into

self.file

after reading it? Does the data need to get operated on together, or can you apply downstream logic to the chunks independently? 3. Does your Metaflow deployment use cloud storage at all, or is everything (flow runs, data storage, etc.) all happening locally?

Sure: 1. In a simple case it might load a large JSONL line by line and do some simple processing and write out to another JSONL. 2. I don't really need the object loaded into self.file. I only want to load it so that the caching process works as intended, but perhaps I'm missing something here. 3. For now it is all local, but will be moved to use a cloud data storage and probably run on the cloud at some point as well.

Assuming everything stays local: • Is there a natural way to split the JSON output written by

function_that_produces_file

? Your original idea about chunking these files on disk so they don't all need to simultaneously be in memory within that function is what I'd start with. If you have a sensible dimension in the data you can split on (chunk N is between dates X_n and Y_n, for example), thats a good place to start. Given the eventual move to cloud storage: • The chunking pattern has huge benefits in this context, especially when downstream processing of the chunks can be done in parallel tasks within in a Metaflow foreach. • We wrote a post about "fast data" with .parquet data, which might be of interest. Lmk if you want to chat on a call sometime about details of your use case how we can apply and extend this pattern for it.