https://outerbounds.com/ logo
Join Slack
Powered by
We've run into an issue causing flows to deadlock ...
# ask-metaflow
s
We've run into an issue causing flows to deadlock when running in Batch: • We have a flow with a 
foreach
step that spawns 20 or so workers. • Each worker gets a ~11GB file from S3 with 
s3.get()
and does some (quick) computation on it. • Some workers finish near-instantly, the others run perpetually (>1h instead of 10 seconds). • Any new flows are stuck in "Starting", then timeout after four minutes. My theory is that workers are competing for the 100GB disk space from the launch template and deadlock while waiting for disk space to free up during the 
s3.get()
call. Is this a likely cause? If so, is there a good way to avoid this situation, aside from preventing the root cause (downloading large files at the start of foreach-ed steps)?
✅ 1
v
I don't know the details of your launch template but presumably every instance gets their own EBS/ephemeral volume, so there shouldn't be competition of that kind
when you say "run perpetually", it implies that they never finish unless you kill them?
are you able to test the flow locally with 
run
, maybe setting 
--max-workers=1
(or other low number) so it doesn't overload your local workstation?
this is to rule out any issues in data/business logic. If it works locally but not on 
@batch
, clearly it is something related to remote tasks
if it only happens on 
@batch
, another thing you can try is to use the new `@batch(use_tmpfs=True)` feature as memory-backed disk space instead of local disks. You'd need to add enough memory (maybe 
@resources(memory=20000)
or more if the file is 11GB) to test it you don't need a custom launch template to use the option, which is a bonus.
if it works with 
tmpfs
without a launch template, it helps to triage the issue to something related to the template
s
Apologies for the delay in responding, long weekend in the UK! The problem only occurs in Batch, not locally. The steps do appear to be actually deadlocked (they run "forever" as in we killed them after two hours when they would normally be expected to run for a minute or so at most). We use the default launch template from the terraform AWS template, but I'm not 100% sure if the 100GB specified by it are per EC2 instance spawned by Batch or per container running on the instance (I have a feeling it is the latter). Using 
tmpfs
would almost certainly solve this problem but be prohibitively expensive for this particular problem. I'll try and make a minimal working example for this and update here shortly - I'm not sure how common this scenario actually is (in our workflows or otherwise), but if the outcome is reproducible, it has pretty nasty results.
v
mysterious. Btw, are the deadlocked tasks in the RUNNABLE or RUNNING state?
it'd be great if you can share a small example. Your setup doesn't sound particularly exotic, so it should work fine as it has done for others in the past 🤔
hi Simon! Did you make progress with this issue?
6 Views
Open in Slack
PreviousNext
Powered by