Hey guys, is it possible to retry flows with

--with retry

so that they are retried only in case of platform/infra errors and not user-level errors? IIRC metaflow retries both in case of user raised exceptions and platform errors, this is good for production but for our staging/test environments we want the flows to fail fast and fail early. If we remove all retries then the issue is that the flow sometimes fail due to infra issues / provisioning timeouts from ECS, etc. that we still want to avoid

yep, the trick is to catch all user code in a

try .. except

block that captures and handles all user-level errors specially. All remaining errors, i.e. platform errors, can be handled as usual. This custom

@platform_retry

decorator should do the trick:

import sys
import time
import traceback
from functools import wraps

from metaflow import FlowSpec, step, retry
from metaflow.exception import METAFLOW_EXIT_DISALLOW_RETRY

def platform_retry(f):
    @wraps(f)
    def wrapper(self):
        try:
            f(self)
        except:
            traceback.print_exc()
            sys.exit(METAFLOW_EXIT_DISALLOW_RETRY)
    return retry(wrapper)

class PlatformRetryFlow(FlowSpec):

    @platform_retry
    @step
    def start(self):
        time.sleep(10)
        print('fail', 1 / 0)
        self.next(self.end)

    @platform_retry
    @step
    def end(self):
        print("done!")

if __name__ == '__main__':
    PlatformRetryFlow()

Ah, wasn’t aware about

METAFLOW_EXIT_DISALLOW_RETRY

exit code. Neat trick, yep that works, thanks!

it’s not part of the public API so technically it may change at some point but it hasn’t for years

Is there a way to do a

platform_retry

on a scheduled Flow on Argo?

good question - that requires a slightly different approach. I’ll create a ticket for it so we can provide a built-in approach that works everywhere

+1 to wanting a platform retry that works on scheduled Argo flows!

noted! https://github.com/Netflix/metaflow/issues/1443 we'll see if we could implement a solution for this in the near term