Hi all, This may be more of a Batch/EC2 question, but how does swap work when running MF steps in AWS batch? We have a step with (very brief) ~35GB memory use. When specifying

@batch(memory=40000)

, the step runs fine. With

@batch(memory=10000, max_swap=40000, swappiness=50)

, we get an out of memory error. Is there anything else that needs to be set up for Batch actually allocating swap or is there a cap on the amount of swap that can be used regardless of the MF decorator?

There's been a couple questions around AWS Batch swap memory usage lately, so here's some extra context to hopefully clear up what's going on: • Docker (which Batch/ECS are using) is not running an OS in your container. It's effectively a running process, and it's realm of concern is different than the underlying OS. Specifically, they're processes in their own namespaces/control groups, but still processes. • Memory virtualization is the OS's concern, not any processes. • Things like swappiness in linux are controlled at the cgroup level, this is what Batch/Docker is exposing to you when you run containers (and via metaflow's decorator). • When you run Batch jobs, those are individual containers provisioned on top of EC2 instances within Batch Compute Environments. You can customize those EC2 instances as part of defining the compute environment, e.g. to use a specific AMI or Launch Template. • In order for containers to be able to use swap memory, you can use an AMI with it preconfigured or you could specify to configure it as part of User Data within EC2 Launch Templates that get run on startup of each instance. Let's say you've attached a Launch Template to your Batch Compute Environment with the following specified in the

User Data

to run on startup, which is a single line to make a 5GB swapfile

#!/bin/bash
fallocate -l 5G /swapfile && chmod 0600 /swapfile && mkswap /swapfile && swapon /swapfile

Once you've updated the compute environment to use that revision of the launch template, you can confirm it's accessible from a example flow:

from metaflow import FlowSpec, batch, step


class SwapTest(FlowSpec):
    @batch(cpu=2, memory=15000, max_swap=25000, swappiness=0)
    @step
    def start(self):
        import psutil

        print(psutil.swap_memory())

        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    SwapTest()

which shows the available swap memory in bytes

sswap(total=5368705024, used=1048576, free=5367656448, percent=0.0, sin=0, sout=90112)

all that said, I'd really caution against trying to lean on swap memory unless it's absolutely necessary and you're also using that launch template to attach a sufficiently configured EBS volume with high throughput / high IOPS provisioned! Using swap memory is not something to take lightly, and it's kinda silly to consider when talking about quite small datasets in the 10s of GBs versus just provisioning more memory. hope that helps! 🙂

Thanks, that's been incredibly helpful!