Hi there! Does anyone have any input on how to best handle sharing of large files between steps, such as videos (sometimes more than 100 gb each) and bigger amount of images in an automatic way. I suppose they wouldn't fit in memory, and as such can't store them directly in

self

. I've stumbled upon metaflow-magicdir, which looks like a fit in principle, but this particular package seems to be stuck in alpha state, and source is no longer available. I've tried using it, but I stumbled upon some issues, assumingly related to its immatureness. Thanks! (I'm new in Metaflow, so excuse me if I'm missing something obvious or don't know all the terms well 🙂 )

S3 or other blob storage is a great place for videos/images to live! When you have lots of data, you likely don't want to rely on local filesystems and instead stream in batches or chunks of data at a time. There's many use cases you won't want to, or it potentially may not even be feasible, to have the entire dataset on locally attached storage. Check out examples of pytorch dataloaders (or higher level huggingface abstractions) for loading images/videos. A good dataloader with multiple workers and a high degree of prefetching will be able to saturate the network interface of large EC2 instances. When done well, you can get comparable or even faster loading performance from S3 than if the files were locally stored (even EBS volumes in AWS are network attached and not really local 😛 ). Those dataloaders can stream in the images in memory for only as long as needed, so you can operate over extremely large datasets in S3. From metaflow's perspective, you wouldn't be persisting the raw data to

self

but you'll likely have pointers/urls to the data which is likely useful to persist as part of the flow

thanks @fresh-laptop-72652! I guess I'll have to write my own logic around this - specifically downloading and uploading (or streaming), creating (unique) s3 urls, storing urls to metaflow artifacts so I can easily resume e.g. failed runs, and such

@narrow-secretary-95110 magicdir worked pretty well in the past for me (it's in the end a pretty simple script if you inspect) - we used it to persist large sets of files (such as those needed by Merlin) in our Metaflow reference implementation of recSys https://github.com/jacopotagliabue/recs-at-resonable-scale/blob/2b16a7701abfad74f9b2c7c3354f1ad27f792137/src/my_merlin_flow.py#L216

thanks @worried-mechanic-36312! with magicdir in particular, when I was resuming a failed run, the magicdir data did not seem to be available in the new run. also, I could not run

extract_magicdir

for the failed run (and did not find another way to access that data). however, maybe I overlooked something also I see that https://github.com/outerbounds/metaflow_magicdir is no longer available