https://outerbounds.com/ logo
Join Slack
Powered by
Hi I got a question regarding the `@timeout` decor...
# ask-metaflow
c
Hi I got a question regarding the 
@timeout
decorator combining with 
@catch
. According to the documentation:
Copy code
It will cause the step to be retried if needed and the exception will be caught by the @catch decorator, if present.
But that is not behaviour I'm observing. When the timeout is reached the whole process is aborted and the pipeline fails.
👀 1
Here's my setup, the flow is triggered with AWS Step-functions:
Copy code
@catch(var="failure")
@batch(gpu=1)
@timeout(minutes=1)
@step
def set_xyz(self):
    ...
Same behaviour was observed with Metaflow 2.18.1, 2.18.11 and 2.19.5.
AWS confirms that the failure is because of the timeout:
Copy code
Status reason
Job attempt duration exceeded timeout
a
@thankful-ambulance-42457 can you help triage
c
Any updates on this?
t
none yet unfortunately, I'll be getting to this today.
@colossal-tent-96436 sorry for the long delay, I finally had a chance to take this for a spin. I'm not seeing any unexpected behavior regarding the decorators on step-functions compared to other executors (batch / kubernetes / argo / local). The description could maybe use some clarification, but it does make mention of combining 
@timeout
with the 
@retry
decorator to achieve task retries. Was your use-case that you want to retry the step if the timeout is reached?
c
Well I do not want to retry, I want the timeout to be caught by 
@catch
decorator and for the pipeline to proceed with the next steps. What I'm observing is that if the timeout is reached the whole pipeline is failing. My use-case is that I'm training multiple models in parallel with 
foreach
and when one of the tasks times-out I want to proceed with publishing all the models that did manage to train.
t
ahh I see. so the timeout and catch are on a foreach step? I'll try that out as well, as my previous test was just on a linear flow
I tried this out with the following flow, and my execution is still reaching the end step as expected. can you give some more details on what is happening during your foreach steps?
catchforeachflow.py
c
I'm not sure what more could I provide. For me same code on local run works well in every scenario (with 
@batch
or without, in simple flow and in foreach) but when executed from AWS Step-functions it fails (I can only do 
@batch
there, but it fails on simple flow as well as on foreach) This is the only log I got: 
Step step_xyz timed out after 0 hours, 1 minutes, 0 seconds
Is there any configuration that may change this behaviour?
t
I'm wondering if this is a resources related race condition, as its proving hard to reproduce without any actual load on the foreach step.. this is what could be happening in rare cases: • the `@timeout`decorator serves two purposes, it sets a max duration for the task execution on 
batch
, and it sets an alert timer during the task execution which raises an exception if reached (this can be caught by 
@catch
) • both the batch timer and the process-level alert have the exact same timeout it is therefore possible that if finalizing the timed out task takes too long, the batch level timeout kicks in and terminates the task
this is something worth trying. It implements a modified 
@timeout
decorator that should not apply the same timeout on the batch/SFN level. ⚠️ The caveat is that if the task process itself gets stuck for whatever reason, then there is no backstop on how long the batch process will run This is meant to simply test out whether the issue is caused by having an identical timeout in the task process and on the batch level. The deco should be a drop-in replacement, just drop the file in your flow folder and change the import to 
from custom_timeout import timeout
custom_timeout.py
alternatively you could try increasing the 
AttemptDurationSeconds
in the Batch job definition by hand to see if that helps.
c
ok thank you, will try it out
23 Views
Open in Slack
PreviousNext
Powered by