I have been struggeling for a while to get anything running on AWS Batch with

gpu=1

. It just stays in the RUNNABLE state. The strange thing is that when I don’t have the GPU constraint, the instance that starts are a beafy p2.8xlarge 🤔 Anyways, Should I be worried that my cluster has “cpu” in the name? (

metaflow-cpu-vccgf4mt20230315100041016700000001_Batch_1e6ca383-5434-303a-b346-2da3fbac1c90

)?

do you know how your Batch compute environment is setup? Probably they include p2 instances since you see them

I recommend not mixing instance class types within a AWS Batch, and having a Job Queue and Compute Environment per each type, i.e., job_queue_gpu and gpu_compute_environment should only be used for GPU workflows and job_queue_mem and mem_compute_environment should only be used for Memory Intensive workflows.

I have set everything up with Terraform using the following. All the instances it added as a desperate attempt to get gpu-steps working. Interesting @curved-island-17262, I though Metaflow would take care of creating however the job queues and compute environments needed. Is that not the case? Should I specify compute environments myself?

# Random suffix for this deployment
resource "random_string" "suffix" {
  length  = 8
  special = false
  upper = false
}

locals {
  resource_prefix = "metaflow"
  resource_suffix = random_string.suffix.result
}

data "aws_availability_zones" "available" {
}


module "metaflow" {
  source = "outerbounds/metaflow/aws"
  version = "0.3.0"

  resource_prefix = local.resource_prefix
  resource_suffix = local.resource_suffix

  enable_step_functions              = false
  subnet1_id                         = "subnet-0bd443a606e8bb7a5"
  subnet2_id                         = "subnet-0d5fbde36dbf0b8cc"
  vpc_cidr_block                     = "172.31.0.0/16"
  vpc_id                             = "vpc-0f863d9c13ac2634f"
  compute_environment_instance_types = [
    "c4.large",
    "c4.xlarge",
    "c4.2xlarge",
    "c4.4xlarge",
    "c4.8xlarge",
    "g4dn.xlarge",
    "g4dn.2xlarge",
    "g4dn.4xlarge",
    "p2.16xlarge",
    "p2.8xlarge",
    "p2.xlarge"
  ]

  tags = {
      "managedBy" = "terraform"
  }
}

# The module will generate a Metaflow config in JSON format, write it to a file
resource "local_file" "metaflow_config" {
  content  = module.metaflow.metaflow_profile_json
  filename = "./metaflow_profile.json"
}