Hi all, Was wondering how the AMI used in the ec2 instance is related to the AWS Batch jobs? I switched to our own internal FAMI for the ec2 instance, and all the AWS Batch jobs get stuck in RUNNING state. However, using the default linux base image works totally fine. If I could understand how the AMI is associated with the AWS Batch jobs, then I can modify the FAMIs, accordingly

nice - good to hear you figured it out quickly 👍

Quick question though - I see an internal AMI for GPUs. Is that necessary for the AWS Batch jobs that require GPUs?

hmm, I have only used the default AMIs with GPUs. There are the usual issues with Cuda drivers etc so I guess you can bake your own GPU images but it takes some effort

So the tf template uses the default AMI with GPU and that's used for both CPU-only AND GPU batch jobs?

@ancient-application-36103 or @average-beach-28850 do you remember the default behavior with the tf template for GPUs?

if I recall correctly our template doesn't specify an AMI, so AWS Bath should automatically pick GPU AMI if you use GPU instances

Okay - thanks Oleg, that makes it a bit tricky for me because I have to use an internal security approved AMI, so not sure how I can tell AWS batch to use a GPU AMI if using GPU instances and CPU AMI for CPU instances. And on that note - I just tried running a CPU-only AWS Batch job with a GPU AMI and that didn't work.

i think you'd need to create two compute environments, one for gpu one for cpu, with custom AMIs

Oh ok so create 2 launch template resources? The template only had a single CPU one but using it seemed to work for both GPU and CPU batch job instances.

yeah I think the setup would be 2 launch templates ,2 compute envs, 2 queues

Okay I'll give it a try, thanks!

let us know how it goes - certainly useful info for others here too

Not sure if this is the optimal solution but I did find an internal AMI that satisfies both CPU and GPU instances. I think this might be easier and more intuitive for the data scientists since they wouldn't have to switch between queues in the

@batch

decorator.