https://outerbounds.com/ logo
Join Slack
Powered by
Question about GPU in batch I deployed the defaul...
# ask-metaflow
c
Question about GPU in batch I deployed the default cloudformation template and got the computer environment. 
c4.large, c4.xlarge, c4.2xlarge, c4.4xlarge, c4.8xlarge
These instances don't have GPUs. Whats the recommended way of getting GPUs in aws batch?
āœ… 1
a
I don't think there is anything too special, it should work if you change the instance type. One thing is to make sure it scales to 0 since GPU instances are so expensive, by setting MinVCPUBatch to 0
c
Forgive my ignorance with cloudformation. Do I modify the template with the instance types I want then redeploy the whole thing? or is there an easier way?
a
there are "parameters" in the template that you can change without editing the template manually
if you deploy the template through the AWS console there is a parameters section somewhere in that UI
āœ… 1
c
Is there a method of updating an existing deployment?
Apologies for these simplistic questions šŸ¤Ŗ
a
yes you should be able to use "update stack" in aws console to change those
c
Great! Thank you
šŸ‘ 1
a
but definitely keep an eye on the compute env size and make sure 
MinVCPUBatch
is set to 0.. very easy to spend a lot of šŸ’° on gpu instances sitting around
c
Thanks I'll do that
you wouldn't happen to have a template with GPU instances as a reference?
a
no but the template YAML would be the same, its just when you deploy it you set those parameters in the AWS console UI to something else than default values
c
Thanks I've set 
MinVCPUBatch
to 0 should I also set 
DesiredVCPUBatch
to 0?
a
yep
c
šŸ™
f
Not sure if AWS Batch finally started taking care of this automatically or not, but you might also need to specify to use the ECS-Optimized Amazon Linux 2 GPU AMI (
AL2_x86_64_GPU
) as part of the EC2 Configuration of the Batch Compute environment If you run into issues, I think it's pretty common for people to split out GPU instance types into their own dedicated compute environment to specify that and also ensure that CPU workloads don't inadvertently get provisioned onto those expensive GPU instances
šŸ™ŒšŸ½ 1
c
@fresh-laptop-72652 Thanks for the heads up. I'm running into issues with it not finding numpy. Do you have a cloudformation template that demonstrates how to do that partition of compute environments.
Copy code
2023-02-22 15:05:18.529 [232/train/1431 (pid 50626)] Essential container in task exited This could be a transient error. Use @retry to retry.
2023-02-22 15:05:18.996 [232/train/1431 (pid 50626)]
2023-02-22 15:05:16.584 [232/train/1431 (pid 50626)] [820bb282-697e-406d-ab23-78824bbc44cb]   File "/metaflow/training_flow.py", line 4, in <module>
2023-02-22 15:05:16.584 [232/train/1431 (pid 50626)] [820bb282-697e-406d-ab23-78824bbc44cb]     import numpy as np
2023-02-22 15:05:16.584 [232/train/1431 (pid 50626)] [820bb282-697e-406d-ab23-78824bbc44cb] ModuleNotFoundError: No module named 'numpy'
2023-02-22 15:05:19.321 [232/train/1431 (pid 50626)] Task failed.
f
hmm my guess would be that's unrelated to AWS Batch and could be due to either ā€¢ not having numpy within a 
@conda
step decorator (or 
@conda_base
flow decorator) to define the python environment ā€¢ if not using one of those decorators, you would need to prebake it into the base python environment of your docker image used in Batch
As for a CFT that shows multiple compute environments, I'm not familiar with one specific to metaflow, but you could update the existing metaflow example with another added: https://github.com/outerbounds/metaflow-tools/blob/master/aws/cloudformation/metaflow-cfn-template.yml#L1386-L1413 Alongside the existing compute environment, you'd want to add another 
ComputeEnvironment
resource, give it another name like 
gpu-...
, specify some GPU instance types like p3/g4dn, if you want it to be on-demand/spot, and so on before also adding it to a Batch job queue Then in your code you could specify it as part of the 
@batch
decorator, e.g. 
@batch(cpu=8, memory=64000, gpu=1, queue="gpu")
here's an AWS sample CFT that shows two compute environments, one for on-demand and another for spot instances, each assigned to different Batch job queues https://github.com/aws-samples/aws-genomics-workflows/blob/master/src/templates/gwfcore/gwfcore-batch.template.yaml that said, if you're just playing around in a dev environment ā€“ personally I'd just create the GPU compute environment in the AWS web UI since it's much easier to tinker with šŸ™‚
just referenced this old gem ā€“ https://github.com/Netflix/metaflow/issues/250 looks like they added the GPU instance types into the existing compute environment and they were good to go! I'd give that a shot first since it'll be a simpler deployment šŸš€
Open in Slack
PreviousNext
Powered by