I am building/maintaining a flow that is part of a core platform, in which we expect two types of transient errors. One originating due to networking issue while building conda environment, another within logic in user code. The user code is very expensive and I don't want to retry if it fails, but if environment creation fails, it should try again. Ex.

from metaflow import FlowSpec, step, conda_base, project


def blackbox():
    # Function not in core library
    import time

    time.sleep(10)

    # maybe fails sometimes
    import random

    if random.random() < 0.01:
        raise ValueError("Gotcha")

    return "SUCCESS"


@project("someproject")
@conda_base(pip=PIP_DEPENDENCIES, python=PYTHON_VERSION)
class SomeFlow(FlowSpec):
    @step
    def start(self):
        self.seeds = range(100)
        self.next(self.heavy_step, foreach="seeds")

    @conda(libraries=MORE_PIP_DEPENDENCIES)
    @step
    def heavy_step(self):
        seed = self.input
        # Do some expensive blackbox step
        # You don't want to retry if this fails due to transient error
        blackbox()

        self.next(self.join)

    @step
    def join(self, inputs):
        self.merge_artifacts(inputs)
        self.next(self.end)

    @step
    def end(self):
        print("ended")


if __name__ == "__main__":
    SomeFlow()

Is that possible with

@retry

? AFAIK it doesn't look like. Would adding a retry in conda_base/conda decorators and passing it all the way to batch_bootstrap fit with the overall design principle of metaflow ? If not, is there a better/easier way to do this ?

afaik, the

@retry

should catch all errors (so the one in blackbox and conda). The retry happens at the very top level (ie: either in the orchestrator or the local runtime). If you want the behavior you describe (retry for conda but not others), you should wrap the code in your step with

try: except:

. This won’t catch everything and you may still retry in that part (due to OOMs for example) but I think it’s close to what you want. In the

except

part, you would have to cleanly terminate your step so that your flow can continue.

yes I am using metaflow extensions. so you mean in the except part I should be doing something (like setting a variable) that can be used in a later step (with no retry) to cleanly exit ?

yep. Metaflow has a fail-fast philosophy so any step that fails will bring down the rest. Here you don’t want to fail but indicate to the rest of the flow: “I failed but want to proceed till the end” (at least that’s what I understood).

hm. in the example above there are two failures, if the blackbox fails, I do want the rest of the flow to fail. it's only the environment creation that shouldn't trigger a full flow failure

ah, sorry, I had misunderstood your requirements.

yep

ooh, that’s going to be a bit tougher. A stupid way would be to add a “check_for_blackbox_failure” step that would check to see if that flag had been set. That step wouldn’t ahve a conda env and would be super dumb so have no retry.

blackbox is a very costly step. If it fails, it usually means some user bug that should involve a developer going through code fixing and re-run. but conda env creation happens rarely, but still enough to require some intervention

ok, for extensions, if you are interested in pip/conda support, the experimental one I was referring to is here: https://github.com/Netflix/metaflow-nflx-extensions.

thanks for the suggestion and taking a look. What do you think about the point I was suggesting in the original post.

that’s an interesting suggestion. So you are basically saying: “don’t retry step but retry env creation” so you would not have a

@retry

on the step but an argument to the

@conda_base

.

going through the code, it seems metaflow creates a job with a clear conda env bootstrap step. can't the decorator itself take a retry or recorvery from failure routine ?

yes, we could do that

Yes. We weren't getting this for most of our flows, until this new workflow that requires 100s of machines spinning up at the same time and some of them having networking issue at bootstrap

cool. Feel free to open a PR with this as well. This seems totally legit.

@high-scooter-88084 are the intermittent conda failures like these in your case? If so, it would be the custom

@conda

implementation DDOSing the upstream channel when you launch multiple parallel workloads

yes they are. Our custom implementation is fairly thin. but you are right it is our implementation that is DDOSing. Before adding something to our custom implementation, I want to make sure this still is in line with the overall principle of how metaflow will develop in the future.

A simple approach here would be to add explicit retries within your custom

@conda

decorator - that way you would also be able to make sure that you are not paying the cost of spinning up yet another AWS Batch job only to retry the environment creation.

Yes, that makes sense. Thanks @square-wire-39606

@ancient-application-36103: yep, I think that was @high-scooter-88084’s idea. I think it may be nice for ours as well but yes, as you say, the caching has made it so that we never had any such issues.

Yep - also the conda package downloads from s3 happen via metaflow.s3 - which already has retries built into it