is there something special that needs to be done to use swap space in Batch? ie if i have a 5TB volume setup in my launch template, do i have to manually mount it or will the 5TB automatically be available?

you want to configure EBS volume to be used as actual memory swap space by the kernel?

yeah, there's a param

max_swap

in the

@batch

decorator

it should work with those Batch params, have you noticed any issues? Couple other considerations about how the storage works and how the EC2 host instance EBS volumes may or may not be exposed to Batch containers running on them: • The Batch/ECS default EBS volume storage is 20GB, which can be modified via Launch Templates to increase storage amount, increase IOPS, change the volume type, and so on. • The Batch/ECS default

ulimit

for the maximum amount of open files/processes per-container is 1024/4096, and each instance defaults to 65536. Typically not a problem for people, but it can surface in unexpected ways. • If you created your Batch compute environment using the Amazon Linux 1 AMI, the

devicemapper

storage driver will be used to preallocate 10GB of per-container storage from the host EC2 instance’s EBS volume that are isolated from each other. In the Amazon Linux 2 AMI, the

Docker overlay2

storage driver is used which exposes all unused space on the root EBS volume to all running containers on the node as a shared filesystem. Even though the Linux 1 AMI was supposed to be deprecated in 2020, Batch kept using it as the default well past then 😬

well, i'm still hitting OOM errors even though i have 5TB of EBS attached in my launch template, so I was wondering if i'm missing some step or there's some hidden limitations surrounding swap usage that I don't know of. So just to confirm, I don't have to manually mount the EBS and setup a swap device?

hm, I'd double check in the Launch Template how you're specifying the EBS volume mapping and if it's being mounted to the root volume of the host instance versus being mounted separately for your respective AMI https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html

i only see one volume in the launch template, so it must be the root? how does batch determine how large to set the swap device though? normally we'd have to create the device using

mkswap

etc?

they're basically just passthrough params to the docker swap memory params https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory depending on the AMI, the root volume is typically either

/dev/xvda

or

/dev/sda1