Hello ! I'm facing a problem with AWS Batch running GPU. When I use

@batch(memory=512, cpu=1, gpu=1)

the job associated remains in

RUNNABLE

state indefinitely : • I have double-checked that compute ressources are sufficient • I have an ECS Cluster with a g4dn.xlarge instance attached to it and launched by the Auto Scalling Group that Metaflow or Batch launched. • I use the AMI

ami-02124cf261ef1e336

so I have cuda installed My CloudTrails shows a RunTask event that fails with response element :

"responseElements": {
        "tasks": [],
        "failures": [
            {
                "arn": "arn:aws:ecs:eu-west-3:*******:container-instance/62da090445a44320811473cd2c0e4055",
                "reason": "RESOURCE:GPU"
            }

It can't understand why, it's like my Batch couldn't see the GPU ressource attached to my EC2 and there is no log to help... When I launch the same step without

gpu=1

it works perfectly well.

Hi! This is likely a capacity or quotas issue. Can you confirm if you are able to create a

g4dn.xlarge

EC2 independent of Metaflow / Batch Compute Environment? For quotas, you can verify under

Service Quotas/AWS services/Amazon Elastic Compute Cloud (Amazon EC2)/Running On-Demand G instances

.

or could be your compute environment doesn't have that instance family supported

Hello, I fixed my problem by changing the AMI 🙂 The cloudformation available in the trainium repo uses

/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id

but it did not work in my environment, I used

aws ssm get-parameter --name /aws/service/ecs/optimized-ami/amazon-linux-2/gpu/recommended

instead

Great! good sleuthing 🙂