https://outerbounds.com/ logo
Join Slack
Powered by
Hello! I have a part of flow completed with no fai...
# ask-metaflow
r
Hello! I have a part of flow completed with no failed steps but the flow itself was marked as failed and stop executing. Does anyone know what might cause this?
✅ 1
v
interesting. Did you start the run locally in a terminal? Do you see any messages on the console?
r
I did. The logs from the terminals seem to just say that there were jobs queued up and nothing after that. I just remembered that we did upgrade the database (changed the instance type) while the flow was executing. Do you know if that is what caused this issue?
v
it shouldn’t and at least you should see some errors if the service was inaccessible
r
hmm icic
v
is it possible that the system was under much load? Something like this could happen if the operating system just decides to kill the process
r
yea I’m not seeing anything dramatic
b
@rich-agent-87730 are you seeing the failed run reflected in the metaflow UI?
r
yes
b
Have you tried refreshing the page? Sometimes, when the heartbeat is not received within a certain time period, they can be marked as failed. It is seen more often with tasks, rather than runs.
r
yea this has been dead for several hours now
I am seeing most of the join steps finished and the flow is dead with a Failed status
b
"most"? Does that mean that some join tasks are shown as failed?
r
none are showed as filed. If I’m expecting 10 join steps, 9 are completed with successes and that’s it
the last one doesn’t show up on the UI and there are no failures
b
Do you see anything interesting in the 
stderr
logs of the tasks that precede that missing 
join
?
r
no nothing interesting
b
Have you run the same flow successfully in the past?
r
This actually is the first time I am running with this particular setting.
oh also when I resumed, it didn’t pick up from that join step but resumed from like 2 steps back
b
Did it finally succeed after the resumption?
v
it sounds like there might have been a task that failed before. It'll resume from the first failed step
r
its a long process, I’m still waiting until it gets to that join step again
well ok sorry, for more context the first run failed at that step, A. the resume finished step A, B, and “failed” on C (the join step) with no failures. when I resumed again, it started from A again instead of C.
b
Can you share the failure reason?
r
the first failure was due to a bug in my code, a 
NameError: name 'variable' is not defined
also this was running on batch
b
Let us know when the latest resume completes
r
will do
this happened again
v
same thing, no errors?
r
yea no errors
v
does it seem to fail at the join step still?
i.e. everything preceding the join seem successful
r
there were some runs in the previous step that wasn’t gotten to
v
one thing you can try is to 
run
(or 
resume
) with 
--max-workers 4
. This reduces the number of parallel tasks that are executed on your workstation/laptop, hence reducing the overall load on the system.
(or even 
--max-workers 1
)
r
oh icic. yea I was using 
--max-workers 800
v
oh ok, that's a lot for a single instance 🙂
r
with a reduced data size though, it ran fine
v
ok, sounds like resource exhaustion then
r
do you know if I can detect that somehow? or what i can look for?
v
are you using a mac laptop or something else?
r
i’m on a big ec2 instance
v
ok. Couple of options: 1. Open another connection to the instance, run 
top
, and keep an eye on the "Mem: Free" column at the top. If it gets very low, that's a problem 2. Use CloudWatch to monitor memory consumption on the instance. In the EC2 console you can enable "high resolution monitoring", so you get more accurate real-time data.
r
oh i am no where near the memory limit of the machine
v
try lowering 
--max-workers
and see if it succeeds. If it does, it is clearly a resource exhaustion issue. If it doesn't, then we can dig elsewhere In general if you want to increase parallelism very high, you should consider using 
@batch
or 
@kubernetes
that give you more resources. It is unlikely that 
--max-workers 800
makes things much faster on a single instance
r
oh sorry, this is running on batch
v
ok, cool. Try lowering it still just to see if it helps, since 
800
is a high number. Besides memory, there are other resources that it may exhaust
r
will do. thank you for your suggestions
👍 1
v
let us know if you can get it running properly. This is useful information for us as well. We can improve the behavior (better error messages!) in the presence of a high 
--max-workers
, if that's the issue
👍 1
4 Views
Open in Slack
PreviousNext
Powered by