Hi everyone, One bottleneck we're currently experiencing is the startup time of EC2 instances on AWS Batch, which significantly slows down our debugging process. I came across this great article that really captures our situation: https://outerbounds.com/blog/faster-cloud-compute. Over 200 seconds of startup time when using a large GPU image from ECR. Unfortunately, we're limited to the open-source version of Metaflow, so we can't use the Outerbounds solution. I saw online that one possible workaround is to bake a custom AMI. Internally, we could create a decorator to automatically re-bake the AMI if the Docker image changes. But before going down that path, I wanted to ask: is there already an existing solution in the open-source version of Metaflow? Or is there perhaps a better way to reduce startup time (maybe through some ECR image caching in aws Batch)? Thanks a lot!

Do you know what is taking the most time. Is it just getting the resources, getting the image, unpacking the environment, etc?

I think it's getting the image, because with the 15GB image, it takes 6 minutes and 30 seconds, but with the 5GB image, it only takes 3 minutes and 20 seconds. Both on the same ec2 ressource

yes - that delay matches our benchmarks and was the motivation for this work. AWS Batch is even more notorious because of the further autoscaling delay that is hard to work around

One approach may also be to use a minimal image and use Metaflow’s environment capabilities. It may not be as fast as prebaked (although if your image load times are that long who knows) but it typically can be pretty quick. It depends what dependencies you need.