Hey all, I have a bit of a paradigm question now: the output of one of our steps is a few large parquet files (probably 5gb in total) in a tree format (there's a folder at the top called data.parquet, which contains files 00.parquet, 01.parquet, etc. Pandas knows how to read the folder as one table). We would like to keep all of these files on s3 and in the same flow output as they come from the same data source. We would also like to be able to read these files by just pointing pandas at the S3 location, rather than downloading everything just to open the tables. What's the best way to store all the data? It seems like it would be to write some helper function to move the entire folder up to S3 using the metaflow.s3 tool, and then save the paths in the artifact, but I want to get an expert opinion on it.

something like this could work