Hey, question regarding the timeout, retry and catch decorators: we have a flow that could run for a minute and can run for 2 days, we want to limit that time for 1 day. i checked the timeout decorator as it seems fit for our use case. in the docs for the timeout i saw that timeout is equal to exception. since we have some issues in our cluster and need sometimes to catch and retry a step (which translate into exception in metaflow). so if i get it right, if i have a step that will run for a day, it will timeout, catch and retry it instead of terminating it. i wonder how i can distinguish between those exceptions and if i can retry based on specific exception (or at least not retry based on specific exception)?

how i can distinguish between those exceptions?

I don't believe the existing catch + retry functionality is aware of exception type. But you can do in the step func code as next answer suggests.

can i retry based on specific exception (or at least not retry based on specific exception)?

You could have try catch block in the step func code. If a specific exception is found, then propagate the error and let metaflow catch + retry do its thing. If another kind of exception is found you can log it and do whatever else e.g., continue or stop the flow execution.

the try and catch block inside the step will not catch cluster issues - which is why we are catching & retrying.

what is the metaflow exception / cluster issue, and when does it occur?