Hi Everyone,
I'm new to Metaflow and have a quick question. If in one of my steps I call a function that produces an output file that might be very large how should I properly handle passing this to the next steps and having it cached properly? I could just load the file into memory and store it as an artifact, but the file could be larger than memory. Alternatively, I could just save the file path as an artifact, but then it seems the caching property wouldn't work properly as the file may change but the file path may stay the same.
Any advice for how to address this? Is it possible to write an artifact in pieces perhaps? Or to set the hash value of an artifact manually?