When setting up an image processing/training job using a huggingface-based model, what's the accepted best practices for loading data into batch instances on aws for repeated experimentation? I have a modestly sized set of images that's roughly 100gb in size, and I want to experiment with various ViT/Swin models to classify a prelabeled image dataset. I can (and have, locally) created a huggingface dataset using their directory structure to assign labels (ie, "(dataset-name)/(train|test|validation)/(label)" as the directory structure. Is the idea behind metaflow.s3 that I should just create that huggingface dataset in my own s3 bucket, then copy it into every instance on creation? (That seems like a lot of time to set up the job, and also kind of a pain if the proper amount of space hasn't been allocated, but I'm not sure if I'm understanding things properly)

@faint-zoo-41552 - where is this data currently stored?

s3

if the data is in s3, then you can use

metaflow.s3

to load the data in your step. it can be faster than using efs. we also just shipped support for efs volumes for aws batch (the release is in progress).

very cool!

relevant thread - https://outerbounds-community.slack.com/archives/C02116BBNTU/p1693953272323469?thread_ts=1693947032.204499&cid=C02116BBNTU To stream in the batches for training/inference, huggingface

Datasets

has lots of great utils and examples of creating iterable datasets/dataloaders from images in S3. By using an iterable dataset you can lazy-load the images on the fly from S3 -- sharding so each dataloader worker will have a different subset of the list of shards to stream and process in parallel. https://huggingface.co/docs/datasets/index