Hello folks, I’m using metaflow with aws batch. I am noticing that steps of different flows are getting executed at same ec2 instance. Is there a way to prevent it?

Any reason why it is a problem? Since they always run on different containers it should not be any different from running on different instances? Is it related to cost analysis or tagging? This is just how AWS Batch is, I am a beginner, but from my understanding - A better solution is to use a different job queue for each flow where each job queue points to different compute environments? or another way is to make it impossible for Batch to satisfy your request by using already existing instance. You could change the compute environment to only have instances of say xlarge - 4vcpus and use @batch(cpus=4), that way only one container can run on an instance. In this case not only flows but also steps run on different instances. Probably first way is better

Thanks for replying.

I hope someone with more knowledge will provide a better solution around the process poll problem. Regarding using large instead of small, I think compute environment spins up machines based on the desired vcpu which would be on the overall demand than just a single batch. In most cases this is good as less startup time, less number of docker downloads etc?

Yeah thats true. But scalling is based on cpus right? When steps are getting spined up in same instance it will cause memory error, right?

You mean the disk storage on a instance wont be enough if too many containers are spun on the same instance? Then in some cases yes, if you are not using custom template for your CE then you only get 30 GB. For many use cases this maybe enough, or if its just less than what you want you can increase vcpu request for the step and then less containers will fit on the machines. Or use a custom template with more disk storage However you might still run into this. So you would want to use instances with cpus in a range. For my usecase I created a launch template with 300GB and added instances with only 4xlarge and 8xlarge.

I was talking about ram memory.

ahh Not sure, since you specify both cpu and ram memory in @batch, I dont think containers with less than requested resource will be created

Valid point. Let me figure out whats the main issue is.