Hi, have a curiosity question for the community who are running metaflow on AWS. when it comes to executing the flow on batch where you are designating the amount of cpu/mem for the batch execution, how have you all optimized the selection of those resource requirements to ensure that ec2 costs to support running the flow are kept in balance maintaining minimal cost to run the flow successfully. Thanks in advance!

@ambitious-bird-15073 @quick-lighter-52296 @User might have some interesting insights here

we are using fargate, but our flows are actually more memory intensive, so not sure the use of fargate is beneficial for us

Is the concern that the user code isn't utilizing the resources made available to the task entirely and you would want to lower the resources allocated to the task?

yeah, but we have flows that can widely vary in resource needs, short of running each with significantly more resources and going back and checking resource usage to fine tune, seeing if any other experiences in the community

@bulky-portugal-95315 I use EC2 instead of Fargate due to it having the flexibility to choose instance classes for specific workloads, under the AWS Batch docs they do recommend using EC2 for workflows that you would typically run on Metaflow. With EC2 you have the ability to select EC2 instances that are optimized for heavy CPU/GPU/MEM usage. For example, these are my current instance classes and the range of values within Staging/Production:

locals {
  instance_classes = {
    staging = {
      GENERAL_PURPOSE = ["m6a.large", "m6a.xlarge", "m6a.2xlarge", "m6a.4xlarge"]
      CPU_OPTIMIZED   = ["c6a.large", "c6a.xlarge", "c6a.2xlarge", "c6a.4xlarge"]
      MEM_OPTIMIZED   = ["r6a.large", "r6a.xlarge", "r6a.2xlarge", "r6a.4xlarge"]
      GPU_OPTIMIZED   = ["p3.2xlarge"]
    }
    production = {
      GENERAL_PURPOSE = ["m6a.large", "m6a.xlarge", "m6a.2xlarge", "m6a.4xlarge"]
      CPU_OPTIMIZED   = ["c6a.large", "c6a.xlarge", "c6a.2xlarge", "c6a.4xlarge"]
      MEM_OPTIMIZED   = ["r6a.large", "r6a.xlarge", "r6a.2xlarge", "r6a.4xlarge"]
      GPU_OPTIMIZED   = ["p3.2xlarge"]
    }
  }
}

For most of my ML workflows, we just need the MEM_OPTIMIZED instance classes(we barely use GPUs’s so the instance above may not be the best to use), the biggest instance class here allows for 128GB RAM if I am not mistaken. With these instances, you also need to ensure that the workflows utilize an optimal ratio of MEMORY/COMPUTE for optimal performance. For example, with the MEMORY_OPTIMIZED instances you should maintain a ratio of 8. In terms of structuring the type of optimized instance classes, I would recommend you have an optimized instance class per AWS Batch Job Queue, i.e., for the MEMORY_OPTIMIZED instance classes they should run on a single AWS Batch Job Queue. If you mix MEMORY_OPTIMIZED and GPU_OPTIMIZED in the same AWS Batch Job Queue it could be the case where for non-GPU tasks it would result in the usage of GPU_OPTIMIZED(special thanks to @fresh-laptop-72652 for telling me about this issue).

@curved-island-17262: this is a huge help, i will definitely take a deeper look into this. our workflows tend to auto select the largest config due to the high memory request set in the resource request of the flow step. will need to take another look through the configs to make sure we’re choosing the best one available between cost and cpu/mem configuration

@bulky-portugal-95315 glad to help. This article will help as well: https://aws.amazon.com/blogs/compute/optimizing-for-cost-availability-and-throughput-by-selecting-your-aws-batch-allocation-strategy/

Cc @calm-smartphone-49719

thanks @User. @bulky-portugal-95315: in Dendra we have so far 1. via inspection picked those instances sizes which give us high cpu utilisation as a % for those workflows that are IO/network constrained 2. re-written some of our code so they can scale up parallelisation based on either memory or CPU limit to achieve maximum resource utilisations on any given instance type. Then we pick the instance type based on our desired turnaround 3. picked instance types with faster network where it makes sense, since sometimes download/uploading data can take almost as long as processing it 4. Batch data across instances/flows to limit EBS volume usage and improve overall turnaround time everyone's data needs are different, these has worked for us Reasonably Well, though by no means perfect

So at least with metaflow step functions integration, the resources are 'hard coded' so you have two options within metaflow within flow: split into separate foreach loops for different memory requirements (ie if you have 100 parallel jobs, but 30 require 100 gb and 70 require 10gb then you create a big memory loop and a small memory loop) outside flow: use the metaflow branches functionality to deploy different versions with different memory specifications ( the

resources

decorator can be fed with a function call)

@plain-baker-2104, the hard coded part in the step decorator is where it feels like we are consistently selecting the same instance type/size. that’s interesting about using the swap space as memory, do you have any articles you’ve been researching on that approach? @curved-island-17262: thanks for that doc, we were using the allocation strategy

BEST_FIT_PROGRESSIVE

, so will look deeper into optimizing that strategy on batch to best select instance types as we were thinking of switching to

BEST_FIT

to see how it changes the instance selection @calm-smartphone-49719: thanks for the insight on how you guys dissected the problem. great insights