https://outerbounds.com/ logo
Join Slack
Powered by
Discovered another issue. We are attempting to run...
# ask-metaflow
a
Discovered another issue. We are attempting to run a for-each with a high level of parallelization, and seems like there is a straggler. It's been stuck in RUNNING state for an hour (should be a fast job - couple minutes) I think this straggler appears to be stuck in the bootstrapping the virtual environment state.
✅ 1
Only one straggler in the fan out:
v
if it's one task out of a wide fanout, it could be a random failure in the container / instance. Setting 
@timeout
and 
@retry
is one way of getting rid of such cases
does it happen every time you run it?
a
Oh yes the 
@timeout
and 
@retry
worked just fine. Thanks Ville!
👍 1
Do you have any feedback though on how to handle corrupt files? These corrupt files take >1h to process, whereas all other files take a couple seconds. In this case 
@timeout
and 
@retry
don't make sense, because we are artificially extending the execution time of the entire pipeline.
v
you could do 
@timeout
and 
@catch
- giving each task a tight time-budget for processing
thankyou 1
👍 1
in a more serious production pipeline you probably want to do 
@retry
too to handle spurious infra issues. Since the timeout happens in the user code in your case, I'd leave 
@retry
on the Metaflow side and handle timeouts in the user code. You could do it e.g. using `multiprocessing.Pool.async_apply` and `wait` results for a few seconds. If the timeout passes, you can set an artifact 
self.corrupt_data = True
to signal that this task didn't complete successfully, which you can then handle in the join step
this way all normal processing happens in a few seconds and infra issues are handled with 
@retry
a
Awesome! Thanks @victorious-lawyer-58417. This is very helpful! 🙏 CC - @sparse-florist-36640
👍 1
5 Views
Open in Slack
PreviousNext
Powered by