Hi all,
We are using docker in our project. Since our image dataset is large (~30-40 GB) the default docker container size of 30 GB is not sufficient for us. We searched for ways to increase this default container size but the
suggested methods are not possible to follow with metaflow. Is there any other method that we can follow to increase this default size? Or is it possible to change the location where metaflow stores these temporary objects to some bigger directory?
Also, we tried using
with S3()
context to get s3 objects as mentioned
here. But the problem here is as temporary files get deleted after the context is over, our program has to fetch all these images again in every epoch. This increases per epoch time significantly (almost 10-20x).