https://outerbounds.com/ logo
Join Slack
Powered by
Are GPUs supposed to work in the vanilla AWS setup...
# ask-metaflow
t
Are GPUs supposed to work in the vanilla AWS setup using Terraform? I have been struggling a lot with jobs that get stuck in the RUNNABLE state. After a whole day if debugging, I have figured out that the AMI that is being used does not seem to support GPUs (
amzn2-ami-ecs-hvm-2.0.20230321-x86_64-ebs
). If I manually copy the instances that gets automatically started by Metaflow and chnage the AMI to 
amzn2-ami-ecs-gpu-hvm-2.0.20230321-x86_64-ebs
and also manually adds these to my cluster, then I can get the flow to run. One other thing that seems strange to me it the name of the cluster is called 
metaflow-cpu-vccg...
(note the cpu in there) I have tried specifying gpu=1 using bot 
@resource(gpu=1)
and 
@batch(gpu=1)
c
not sure on any specific Terraform template, but iirc the default one does not have GPUs. if you go to the AWS batch dashboard, it shows a compute environments table where you can double check what instance types your job queues can make resources from. if you see something like 
p2
, 
p3
, 
p4
in that cell of the table you should be good to go. but if there are no gpu instances and do 
@batch(gpu=N)
i've seen Metaflow hang in that RUNNABLE state, which makes it kind of tricky to know what is (not) happening.
w
+1 @crooked-jordan-29960 - if you use defaults, the machines available do not have GPU and yuo have to manually edit the cloud formation to allow them (note: sometime AWS dont have them for you if the account is new, and you have to ask for more compute through AWS limit requests)
@ancient-application-36103 -> feature requests: include standard p (at elast the p2?) in the default CF template for AWS?
đź‘Ť 1
a
would love to - the only concern is the bill that folks may inadvertently accrue if they are not careful. We can definitely have another GPU specific CF template tho!
w
I may be wrong, but as far as I know, if you don't request a batch gpu, AWS will not use a p machine
(so it falls back to g or whatever is the others in the template)
a
Yeah - but if you don't set your min vcpu to 0, then there might be idle instances lying around, incurring a steep bill.
moneys 1
🙌🏽 1
🙌 3
đź’° 1
t
Thanks for the quick replies everybody. I should have told you that I have plenty og GPU instances in the cluster, and Metaflow/Batch correctly chooses GPU instances (those are the ones I copy). This is a problem with the AMI and the ECS agent not supporting GPUs.
Copy code
# Random suffix for this deployment
resource "random_string" "suffix" {
  length  = 8
  special = false
  upper = false
}

locals {
  resource_prefix = "metaflow"
  resource_suffix = random_string.suffix.result
}

data "aws_availability_zones" "available" {
}


module "metaflow" {
  source = "outerbounds/metaflow/aws"
  version = "0.3.0"

  resource_prefix = local.resource_prefix
  resource_suffix = local.resource_suffix

  enable_step_functions              = false
  subnet1_id                         = "subnet-0bd443a606e8bb6d8"
  subnet2_id                         = "subnet-0d5fbde36dbf0bda7"
  vpc_cidr_block                     = "172.31.0.0/16"
  vpc_id                             = "vpc-0f863d9c13ac2696b"
  compute_environment_instance_types = [
    "c4.large",
    "c4.xlarge",
    "c4.2xlarge",
    "c4.4xlarge",
    "c4.8xlarge",
    "p3.2xlarge",
    "g4dn.xlarge",
    "g4dn.2xlarge",
    "g4dn.4xlarge"
  ]

  tags = {
      "managedBy" = "terraform"
      "belongsTo" = "metaflow"
  }
}

# The module will generate a Metaflow config in JSON format, write it to a file
resource "local_file" "metaflow_config" {
  content  = module.metaflow.metaflow_profile_json
  filename = "./metaflow_profile.json"
}
Would there usually be one or two compute environments? The name 
metaflow-cpu-vccg...
smells a little like there are supposed to also be a 
metaflow-gpu-vccg...
compute environment?
a
in this scenario I'd use tf submodules individually, have two instances of https://registry.terraform.io/modules/outerbounds/metaflow/aws/latest/submodules/computation one for GPU and one for CPU
đź‘Ť 1
and choose one of them via metaflow config
t
Is that the recommended approach? So I should switch between the configs every time I add the 
--with batch
flag? @average-beach-28850 Also, I don’t understand why this would make any difference. What would make the instances suddenly support GPUs? Or phrased differently, what would be the setting/input that I would set different than the settings I have today? There seems to be no settings/input to that submodule which controlles whether the cluster is a “gpu cluster” and it also seems like I cannot decide which AMI to use. So I would still have the same issue, right?
a
AWS Batch has some special heuristic to auto select GPU ami based on compute env settings https://docs.aws.amazon.com/batch/latest/userguide/batch-gpu-ami.html , if you don’t explicitly ask for a specific AMI. I’m not sure what’s the behavior for envs with mixed instance types. But I remember it working at least if you use a pure gpu env
You’d switch between configs, yes. there is also a way to specify Batch queue in @batch decorator parameters per step
t
@average-beach-28850 Am I understanding you correctly that having mixed instances in the cluster causes the AWS heuristics to select the wrong AMI. Damn, thats tricky. I will try with a GPU also cluster and see if that solves it. Thanks for pointing this out.
a
yep I remember removing the AMI from our launch template to let AWS decide https://github.com/outerbounds/terraform-aws-metaflow/pull/30
🙌 1
t
Okay. The launch template is still called “cpu” though. Am I supposed to define my own launch template I wonder? On another note, it looks like this file is not used anymore: https://github.com/outerbounds/terraform-aws-metaflow/blob/master/modules/computation/data.tf
FYI. I just tried with a completely fresh setup (terraform destroy -> terraform apply), this time using only 3 different 
g4dn
instance types. I still see a cluster called something with cpu: 
metaflow-cpu-pn0a....
And the launch template still uses a non-gpu supporting ami.
10 Views
Open in Slack
PreviousNext
Powered by