https://outerbounds.com/ logo
Join Slack
Powered by
Hey guys, I have a metaflow step with these resour...
# ask-metaflow
c
Hey guys, I have a metaflow step with these resources: 
@resources(cpu=16, memory=32768)
and these 
<http://vars.tf|vars.tf>
:
Copy code
variable "batch_type" {
  type        = string
  description = "AWS Batch Compute Type ('ec2', 'fargate')"
  default     = "ec2"
}

variable "compute_environment_desired_vcpus" {
  type        = number
  description = "Desired Starting VCPUs for Batch Compute Environment [0-16] for EC2 Batch Compute Environment (ignored for Fargate)"
  default     = 8
}

variable "compute_environment_instance_types" {
  type        = list(string)
  description = "The instance types for the compute environment"
  default     = ["c5.large", "c5.xlarge", "c5.2xlarge", "c5.4xlarge", "c5.9xlarge"]
}

variable "compute_environment_max_vcpus" {
  type        = number
  description = "Maximum VCPUs for Batch Compute Environment [16-96]"
  default     = 64
}

variable "compute_environment_min_vcpus" {
  type        = number
  description = "Minimum VCPUs for Batch Compute Environment [0-16] for EC2 Batch Compute Environment (ignored for Fargate)"
  default     = 8
}
however, the aws batch job for this step keeps being assigned with 
CPU=1 Memory=4GB
do you know why?
a
what does your flow look like? curious about the decorators for each step
c
the step in question has this decorator:
it resulted in this job definition and, as you can see, it only has 1 cpu and 4 gb
a
How is this workload getting deployed? python flow.py run —with batch?
c
yes exactly
a
The job definition will have 1 CPU and 4G RAM. The actual job will have the right CPU and RAM allocated. We share job definitions across multiple workloads
What do you see for the actual job?
c
Copy code
INFO:__main__:CPU Cores: 18
INFO:__main__:Memory: 68.67889022827148 GB
however, these were the logs from the step
so I you are right
a
Those values from the code are going to be the same regardless of the resources you provide. psutil returns node metrics and not container metrics
🙌🏽 1
🙌 1
c
but, on a different topic, is it possible the role I’m using has some limitation (at least on the number of batch jobs)? because the pipeline is set to have a 
max_workers
of 20 but it can’t have more than 1 instance at the same time with those resources. all the others are stuck in 
runnable
a
If you go the AWS batch jobs dashboard and try to locate the job, you should see the correct resources allocated to it
The resources you have specified can only be found in c5.9x large instance and only one job will fit inside that instance. However, you are capped at only running one of those instances - since you specify a max of 64 vcpus and each c5.9x instance has 36 vcpus. If you lower your memory and cpu requirement slightly then you should be able to fit your workload on a c5.4xlarge (not all the resources advertised by AWS are available on the instance - usually it’s slightly less) and you can run 4 such instances with the current configuration of your instance pool.
c
got it, that’s precious information, thank you!
16 Views
Open in Slack
PreviousNext
Powered by