Hi, we're using a custom docker image for setting up our environment. Model runs for one epoch, but at the end for validation when it tries to load images. We get "No Space Left on Device Error". We have enough disk space and memory allocated for the experiment. Suprising it works fine on our on-prem compute facilities.

Hey Harsh 👋 Are you running locally or using

@batch

/

@kubernetes

? If local, are you using a the local datastore and it's trying to persist the metaflow artifacts locally? Could you also clarify "when it tries to load images" and what that entails – if I were to guess, it sounds like this is unrelated to metaflow specifically and could be from other code trying to inadvertently download/store image files locally.

Hi @fresh-laptop-72652, we're using @batch . So basically, "loading images" in our dataloaders working to load images from s3 bucket. We found the bug to be in S3() object. It creates temporary copies while fetching the objects. I saw somewhere in the discussion where Vile mentioned while using,

with S3() as s3: 
     s3.get()

It takes care of the local copy creation and deletion. We tried that and it solved our issue.

Glad to hear that's resolved 🙂 For managing disk space on AWS Batch jobs, you'll likely want to create a custom EC2 Launch Template with the bigger EBS volume sizes specified, which can then be attached onto the Batch Compute Environment

Thank you guys.

Is there anyway to integrate it as part of

python sample_flow.py step-functions create

? Or you have to wrap it in another script and look for the EC2 Launch Template after the step function is created for modification?

you could create a script that modifies the launch template when a failure is detected and resubmits the workload