I'm confused how/when Batch decides to spin up a new EC2 instance vs reusing the same one. In one of my steps I requested

cpu=64, memory=384_000, gpu=1

and it's correctly running that task on a

g3.16xlarge

instance (which has 64 vCPU, 488 GiB memory and 4 GPUs). I am expecting no other tasks to be scheduled on that instance since the core count is maxed out, but I see another 24 tasks (with

cpu=1, memory=4

) running on the same instance. What is going on?

that shouldn't be the case - were all the 25 tasks running at the same time on that instance?

yea.. it's confusing

the instance with 23 tasks has available CPU shares

i went through the tasks and found the larger one.. it says it's running. I'm not sure why it's showing the memory as 250gb though:

can you locate this job from the AWS Batch console?

yup i see it in the batch console

and what are the resources set to?

image.png

did you set the memory requirement through the decorator or through CLI?

via the decorator.. although it's possible the code i'm looking at was updated and the SFNs weren't updated accordingly

the only other smoking gun would be the actual instance type that's running

Ok my bad, I didn't filter the tasks to the instanceId properly. Looks like what's happening is, after the large task finishes on the g3.16xlarge instance, all the smaller tasks get allocated on there before the instance terminates.