Hi all, We are using docker in our project. Since our image dataset is large (~30-40 GB) the default docker container size of 30 GB is not sufficient for us. We searched for ways to increase this default container size but the suggested methods are not possible to follow with metaflow. Is there any other method that we can follow to increase this default size? Or is it possible to change the location where metaflow stores these temporary objects to some bigger directory? Also, we tried using

with S3()

context to get s3 objects as mentioned here. But the problem here is as temporary files get deleted after the context is over, our program has to fetch all these images again in every epoch. This increases per epoch time significantly (almost 10-20x).

hi Chandan 👋 firstly, if you use the S3 client without a context manager (without

with

), it won't delete temporary files, like here:

s3 = S3()
s3.get_many(...)

regarding allocated disk space, are you using

@batch

or

@kubernetes

?

@batch

unfortunately there isn’t a straightforward way to increase available disk space for

@batch

, which is a limitation on the AWS side. You have to create a custom Launch Template and a new Compute Environment like described here https://aws.amazon.com/premiumsupport/knowledge-center/batch-job-failure-disk-space/

yup sounds like you'll want to create a custom EC2 Launch Template with the bigger EBS volume sizes specified, which can then be attached onto the Batch Compute Environment couple other thoughts: • S3 is a great place for storing image data and for nearly all real-world use cases you won't want to, or it potentially may not even be feasible, to have the entire image dataset on locally attached storage. • A good dataloader with multiple workers and a high degree of prefetching will be able to saturate the network interface of large EC2 instances. When done well, you can get comparable image loading performance from S3 as if the files were locally stored (even EBS volumes in AWS are network attached and not really local 😛 ). Those dataloaders can keep the images in memory for only as long as needed, so you can operate over extremely large datasets in S3. • (Super overkill and absolutely not necessary here – but mentioning for educational purposes) There’s also the HPC side of the spectrum where you might go down the path of mounting EFS/FSx volumes with your images already saved and well structured (via the Batch Launch Template). tl;dr make a launch template to expose more storage to your batch jobs, and when in doubt use more dataloader workers for each GPU and a higher degree of prefetching per worker to maximize S3 throughput.

firstly, if you use the S3 client without a context manager (without

with

), it won’t delete temporary files, like here:

One question here: Suppose in the first epoch I have fetched all the images and now they are all stored at my tmproot directory as temporary files. In the subsequent epochs when I again fetch those same images (using s3.get(“s3_url”)) does it pick up those already saved temporary files or it again downloads those images and saves as new temporary files? I expected that former would happen but actually latter is happening. Here the S3 class I am using is S3(tmproot=“/dev/shm/“). In the docker container /dev/shm is mapped to shared_memory which we specify through @batch.

the S3 client doesn’t do any local caching - it fetches the files again every time you call it If you can fit your whole dataset in memory, by far the fastest approach is to download them first before training starts and then use the downloaded files across epochs. Using /dev/shm shouldn’t be necessary. If you have enough memory on the instance, files stay in memory (disk cache) automatically