https://outerbounds.com/ logo
Join Slack
Powered by
Hello fellow Metaflow users, Could I please kindl...
# ask-metaflow
a
Hello fellow Metaflow users, Could I please kindly survey how long it takes you to provision an AWS Batch GPU instance? For us, it's about 8 minutes overhead (the time elapsed from API runtime call to Batch Instance status transitioning to "STARTING") and it looks like most of that time is due to EC2 initialization. We suspect it's because we are using an internal security hardened AMI that has a lot of bloat. We are trying to think of ways to reduce that overhead because 8 minutes to start up a batch instance is quite a long time and some of our users are finding that a bit frustrating when debugging. I can clarify that we set min cpus and desired cpus to both zero for the launch template
✅ 1
v
what instance type? I presume you are talking about on demand (not spot)
could you test with a vanilla AMI to see if it’s about your AMI or not?
a
Yup on-demand and it was 
p2.xlarge
, but don't think it matters because 
m1.small
also takes a long time.
Yup I plan to do that to identify the root cause of it. I also plan to create multiple batch queues as well which will then allow us to use different AMIs - one GPU optimized and the other for CPU workloads. Right now I'm using a single AMI that's GPU optimized. However, that still wouldn't solve the problem of the GPU instances having 8 min overhead
v
re: GPUs - some instances (like 
p4
) are just impossible to get right now, so wait times can approach infinity. Some other instances can take a while too, depending on the region. If you experience slowness with 
m1
too, it sounds it’s some other issue though
also is it 8mins always or 8mins when spinning up new EC2 instances in the compute environment? Cold start latencies are naturally higher, although 8mins sounds a bit long
launching a job on an existing available instance should take <1min
a
Oh yeah launching a job on existing available instances is definitely seconds. It's just when spinning up new EC2 instances in the compute environment which is usually always because we set min/desired vcpus to zero for the batch compute environment
v
ok. Lmk if you still experience slowness with a standard AMI
👌 1
a
Confirmed that the standard AMI takes about 1.5 minutes to get to STARTING.
Ok we were able to get the Batch job initialization using hardened AMI down to about 1.5 minutes now :)
v
nice!
Open in Slack
PreviousNext
Powered by