Hi. I’m trying to set up a flow that uses both Far...
# ask-metaflow
s
Hi. I’m trying to set up a flow that uses both Fargate and GPU-enabled EC2 instances in AWS Batch. Some steps would be run on a default queue with Fargate, while others steps would run with GPUs on a different EC2-based queue. However, specifying GPU and queue ARN with the
@batch
decorator throws this error:
Copy code
"An error occurred (ClientException) when calling the SubmitJob operation: GPU is not applicable for Fargate."
Any advice on handling mixed Fargate and GPU-based EC2 steps in the same flow?
āœ… 1
g
It looks like you're pointing to the CPU (fargate) queue and requesting GPU capability. Try and setting the queue name to your GPU queue. Perhaps you can try and setup a queue with CPU/Fargate compute env at prio 0 and GPU/EC2 with prio 1? And the
@batch
or
@resource
will pick up the appropriate compute env? Alternatively you could setup 2 separate queues and set the appropriate queue name for each step
s
thanks. There is already two separate queue for the two compute environment and the step is calling the correct GPU queue but I still get this error. Am I doing anything else wrong?
g
The fargate error makes me think you're calling the GPU step against the CPU queue. How do you set the step'squeues in your code? Can you double check the CPU named queue is associated with fargate and the GPU is associated with ec2? And remember that fargate can't do GPU
šŸ‘€ 1
s
Thank you for the help. It was solved.
šŸ™Œ 1