Hey friends, not strictly a Metaflow question, but some of you smart people must be working with Huggingface models in your flows. We're seeing unreasonably slow model load times from AWS EBS volumes with 7-20B parameter models. As we didn't want to download the models from HF at every run, we store them on the volume and use

AutoModel.from_pretrained("path/to/model", local_files_only=True

) to load the models when needed. But it takes ages, and what is odd is that downloading (which also writes to cache) + loading the model to memory on the spot seems to be faster, so I'm not willing to blame disk speed... soo anybpdy seen something similar? :)

Have you tried the most recent release with

tmpfs

support - https://github.com/Netflix/metaflow/releases/tag/2.8.3#features.1

nope, thanks for the pointer!

more docs coming soon! Note that loading from S3 with

metaflow.S3

- which now works seamlessly with

tmpfs

if you enable it - can be (much) faster than loading from local disk / EBS

I can actually reproduce this outside of Metaflow so I'm thinking i'm doing something silly with the hf package, but this looks like it's worth looking into

interesting. Let me know what you find. You can test IO without HF to confirm that the issue is (not) specific to HF