https://outerbounds.com/ logo
Join Slack
Powered by
Is the performance on AWS Batch supposed to be com...
# ask-metaflow
w
Is the performance on AWS Batch supposed to be comparable to raw EC2? I'm seeing my jobs take significantly (at least 3x) longer on Batch, on similarly specced resources. Any tips/tricks to debug this?
✅ 1
a
it should be the same EC2 instance. Are you configuring any launch templates or custom AMIs?
w
I have a launch template to increase the EBS, but that's the only difference. I am trying to use swap though, if that matters
f
ECR vs AMI - which is better for Metaflow @ AWS ?
v
there’s some AMI at the bottom always even if you pull images from ECR, but it’s certainly easier to add custom dependencies in images vs a custom AMI
@wooden-dusk-90720 are you sure it is the same instance type on Batch vs your dev workstation? Even if 
@resources
are the same, the instance type can make a difference due tenancy behavior, CPU type etc
if you have exactly the same instance type and tasks using the full instance, @batch should show the same performance
note that Batch can run multiple tasks on the same instance if your 
@resources
are low enough, which is good for cost efficiency but it can make tasks slower. Increase 
@resources
to match the instance size more closely to get rid of the noisy neighbor effect
w
Thanks, I managed to figure it out. I had to increase 
shared_memory
because the default (64mb) on Docker is different to EC2 (half the RAM)
👍 1
By the way, is there a way to force Batch to use a different instance each time, without increasing 
@resources
?
v
you can set up multiple compute environments with different instance types and target them with 
@batch(queue=…)
✅ 1
4 Views
Open in Slack
PreviousNext
Powered by