Hi Team, I have created the argo template from the meta flow file and deployed it into Kubernetes with s3 as my datastore, now I am looking a way to resume this flow in case run fails on any step. How can I pass this run id to resume it with argo.

you can resume Argo executions on the command line as explained here

So, retry would try to resume at least the current executing pipeline. right? Are we planning to support resume in argo templates in future.

you can choose which run you want to resume - you can resume any run locally

@straight-shampoo-11124 does airflow support resume? locally orchestrated resume seems very fragile for large production workflows. any internet dropout will cause the run to fail also the orchestration surprisingly seems to require a lot of resources [What resources are being used when using local orchestration? I understand we could use a cloud instance, but it would help to know how to spec the machine ] we have had resume runs fail because of insufficient file handles. running locally ( just the orchestrator) with batch executing the jobs seemed to require memory/cpu too

yep, note that the idea with Metaflow's

resume

is not so much to complete a failed production run but rather make it possible to triage, reproduce, and fix the issue locally, and then push a fixed version back to production. That's why

resume

create a new run in the user's own namespace - it doesn't try to pretend that the error didn't exist in the production run

So I assume original OP (and others previously) are looking to resume production runs. So I feel it might be helpful to spell out more clearly best practises on how to handle these issues - eg rewriting the code more fault tolerantly. as discussed before (https://outerbounds-community.slack.com/archives/C02116BBNTU/p1625224662381200), this feels like something metaflow should enable, rather than each user coming up with their own resume functionality by checking that the step has not previously been completed) we use

@retry

but it doesn't support conditions ( eg retry if docker time out error but not if OOM or syntax error) OOM - this is painful because the memory spec is 'hard coded' in eg the

@resource

decorator- it would be helpful to eg retry with increased memory ( again on condition of OOM error)

yep, I hear you. There are certainly valid use cases for resuming production runs. I linked this thread to a Github ticket about the issue

In terms of "resuming from failed step" this (https://outerbounds-community.slack.com/archives/C02116BBNTU/p1676403121403509) was the kind of thing I was thinking of in terms of "rewriting the code more fault tolerantly". @straight-shampoo-11124 very excited this is nearing release. Rather than "resuming production runs", a "production run" should be composed of sub-flows, and each sub-flow can be rerun...

@straight-shampoo-11124 hope you don't mind if I resurrect this thread 😄 We have a similar question come up about resuming runs but using argo as the execution engine. The concrete use-case for us is data scientists want to run lots of one-off experiments using metaflow, and they rely on argo to orchestrate long-running tasks. The runs largely consist of data preparation -> model training, but a convenience feature is to reuse existing datasets on runs if you are only tweaking the latter part of the pipeline. When running locally, they can run something like

.. flow.py resume train

to skip the data prep step, but this only works for small datasets where runtime is fast. As a workaround currently, we add an extra parameter so that we can pass a run_id to argo, then in our data-prep step we skip logic and manually fetch artifacts over client API using passed run_id if it is given. This is fine but seems like reinventing the wheel a bit, when resume already does this so well when not using argo

makes sense. Skipping based on existing data is not a bad pattern. Another pattern to consider is to use event triggering: have data prep in one flow and the rest in another, linked via

@trigger_on_finish

if you want to skip data processing, just run the second flow

👍 So do you think it would ever make sense to port

resume

to work with argo as execution engine somehow? Otherwise we will focus on these alternative approaches which require a bit more flow engineering

yep, the request makes sense for sure - not the first time the topic comes up. Can you please upvote / comment this issue that captures the feature request - thanks!