Hi my production AWS ECS cluster was facing issues with a

DockerTimeoutError

yesterday, our AWS EBS Burst Balance reached 0%(image 1). Based on this thread and a couple others I did the following: 1. Set METAFLOW_DEFAULT_CONTAINER_REGISTRY to

public.ecr.aws/docker/library/

2. Move from GP2 to GP3 with the following custom AWS Launch Template(AWS EBS):

ebs {
      volume_size           = 100
      delete_on_termination = true
      encrypted             = true
      volume_type           = "gp3"
      iops                  = 3000
      throughput            = 125
    }

But I am still noticing large StorageWriteBytes that are probably going to cause more issues from a single pipeline(image 2). Here is a bit of the code for the pipeline:

class Flow(FlowSpec):
    @step
    def start(self):
        self.next(self.process_video, foreach="chunks")

    @resources(memory=8_000)
    @pip(
        libraries={
            "mux-python": "3.7.1",
            "opencv-python-headless": "4.6.0.66",
            "openai": "0.27.0",
            "scikit-learn": "1.2.1",
        }
    )
    @step
    def process_video(self):
        self.next(self.join)

    @step
    def join(self, inputs):
        self.next(self.extract_keywords)

    @resources(memory=32_000, cpu=4)
    @pip(
        libraries={
            "mux-python": "3.7.1",
            "opencv-python-headless": "4.6.0.66",
            "wheel": "0.38.4",
            "setuptools": "65.6.3",
            "spacy": "3.4.3",
            "nltk": "3.7",
            "keybert": "0.7.0",
            "pytextrank": "3.2.4",
            "rake_nltk": "1.0.6",
            "yake": "0.4.8",
        }
    )
    @download_nlp_libraries() 
    @step
    def extract_keywords(self):
        self.next(self.end)

I noticed the owner of the Flow using

@pip

more than

@conda

even extending it to create a

@download_nlp_libraries

which basically installs Spacy and NLTK extentions such as

"stopwords", "punkt", "averaged_perceptron_tagger", "wordnet", "omw-1.4"

. The pipeline does a lot of processing on videos and text data leveraging 3rd party APIs. I am struggling to understand what could be writing so much to storage? Is it the usage of the NLP libraries via

@pip

especially since its done within a

foreach

? Could creating a docker image and using it be a solution to this?

you can try increase

@resources

to force placement on distinct nodes to see if it helps

couple things standing out: • the ECR public mirror is only going to help if you're getting throttled by the dockerhub repo, that said if you're in AWS you should be using that repo anyways for public images or better, point to a private ECR repo within your AWS account (can make copies of public images, build your own with custom deps, etc) • the docker images will still need to be downloaded onto the EC2 instances provisioned in your Batch Compute Environments regardless of which repo they come from ◦ depending how your ECS cluster is configured, you may also be pulling the docker images repeatedly (even if its the same base image) per job that's colocated – see the

ECS_IMAGE_PULL_BEHAVIOR

of https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html ◦ you can tweak those params via the launch template user data or a custom AMI • its hard to be prescriptive on EBS volume configuration without knowing the specifics of your business and use-cases, however I would say that 3k IOPS/125MiB/s is on the low side (it's the minimum after all 😛 ). Without having more specifics, 6k IOPS/500MiB/s is good while maintaining a similar cost in batch workloads – 100GB volume size is a bit on the low side too, but again really just depends on your use-cases. • Batch Compute Environments don't get new versions of Launch Templates automatically (last I checked at least). Even if you specify to use

$Default

or

$Latest

in Batch and update that pointer to the Launch Template, you will still need to modify each of the compute environments for the new version to take effect. It used to be you needed to recreate them from scratch, fortunately a year or two ago Batch made an update so now they'll get reapplied when any modifications are made. You can go to the Batch Compute Environments in the AWS web console, select modify, change nothing, hit save, and the new template will then take effect after the underlying ECS cluster is modified. This is a major gotcha, so any time you make changes to your Launch Template, verify the changes took effect on the EC2 instances provisioned within your compute environments (e.g. inspect the attached EBS volumes for your new config) • downloading those packages from

pip

each time is wasteful network/disk IO. I'd also ensure that large pretrained models aren't being unnecessarily downloaded each time

@fresh-laptop-72652

the docker images will still need to be downloaded onto the EC2 instances provisioned in your Batch Compute Environments regardless of which repo they come from

depending how your ECS cluster is configured, you may also be pulling the docker images repeatedly (even if its the same base image) per job that’s colocated – see the

ECS_IMAGE_PULL_BEHAVIOR

of https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html

you can tweak those params via the launch template user data or a custom AMI

So far we are not leveraging Docker images yet for our workflows(I am going to start enabling access to our AWS ECR repos) and solely using the base image that Metaflow uses, the Python image. If I were to update the

ECS_IMAGE_PULL_BEHAVIOR

to

prefer-cached, would it just use the same set of images,

which in this case are the Python versions would I then not be pulling and building the images but instead just using the cached images?

its hard to be prescriptive on EBS volume configuration without knowing the specifics of your business and use-cases, however I would say that 3k IOPS/125MiB/s is on the low side (it’s the minimum after all 😛 ). Without having more specifics, 6k IOPS/500MiB/s is good while maintaining a similar cost in batch workloads – 100GB volume size is a bit on the low side too, but again really just depends on your use-cases.

For the volume size, is that setting the max size per instance that is allocated or is it for the entire AWS ECS cluster? After consulting my DevOps team, I will bump the IOPS and volume size.

Batch Compute Environments don’t get new versions of Launch Templates automatically (last I checked at least). Even if you specify to use

$Default

or

$Latest

in Batch and update that pointer to the Launch Template, you will still need to modify each of the compute environments for the new version to take effect. It used to be you needed to recreate them from scratch, fortunately a year or two ago Batch made an update so now they’ll get reapplied when any modifications are made. You can go to the Batch Compute Environments in the AWS web console, select modify, change nothing, hit save, and the new template will then take effect after the underlying ECS cluster is modified. This is a major gotcha, so any time you make changes to your Launch Template, verify the changes took effect on the EC2 instances provisioned within your compute environments (e.g. inspect the attached EBS volumes for your new config)

I use terraform for my Batch CE, updating the CE just requires me to force a change of state within the creation of said Batch CEs.

downloading those packages from

pip

each time is wasteful network/disk IO. I’d also ensure that large pretrained models aren’t being unnecessarily downloaded each time

After further investigation this was the actual issue, one step is downloading multiple versions of BERT via different libraries and it coincides with the massive increase in

StorageWriteBytes

. I was thinking about creating a custom Docker image for this step which loads all the necessary dependencies but I am pretty sure it’s going to be quite large, is

StorageWriteBytes

still impacted by the pulling of large Docker images or is it just being impacted cause we are installing it within the step via

@pip

which does not cache the dependencies as in

@conda

? Also, note that some of the packages to install aren’t on conda.

• the ECS instances provisioned by Batch would still need to pull a unique image at least once and keep in local storage, caching is just potentially reducing overhead of repulling the same image when multiple jobs are colocated on the same instance and using the same image • the EBS volumes provisioned onto those ECS/Batch instances are per-instance (Launch Templates operate at the instance level) • I'm not aware if terraform will ensure references to Launch Templates are updated when it's a pointer like

$Default

when there are no other changes to Batch CE – I would highly recommend confirming your expected changes are applied afterwards. • whether or not large model artifacts are prebaked into docker images, cached in S3, or downloaded from public repos at runtime – you will still end up with a copy of the model in local storage. Prebaking artifacts into the docker image is more useful for performance, as you'll only be limited by download speeds from your ECR, and once a container is run it's already ready to go – in comparison, having to download large model artifacts over the public internet at runtime can have many potential chokepoints/throttling/ingress concerns.

the ECS instances provisioned by Batch would still need to pull a unique image at least once and keep in local storage, caching is just potentially reducing overhead of repulling the same image when multiple jobs are colocated on the same instance and using the same image

I see this will definitely help out, especially for our jobs running in parallel on the same instance.

I’m not aware if terraform will ensure references to Launch Templates are updated when it’s a pointer like

$Default

when there are no other changes to Batch CE – I would highly recommend confirming your expected changes are applied afterwards.

Will confirm this, there is an argument when creating the

aws_batch_compute_environment

to specify the version of the launch template used.

whether or not large model artifacts are prebaked into docker images, cached in S3, or downloaded from public repos at runtime – you will still end up with a copy of the model in local storage. Prebaking artifacts into the docker image is more useful for performance, as you’ll only be limited by download speeds from your ECR, and once a container is run it’s already ready to go – in comparison, having to download large model artifacts over the public internet at runtime can have many potential chokepoints/throttling/ingress concerns.

Great then I will definitely need to work on this! @fresh-laptop-72652 can’t thank you enough for all the help