https://outerbounds.com/ logo
Join Slack
Powered by
We are deploying via argo-workflows, and there sev...
# ask-metaflow
c
We are deploying via argo-workflows, and there several things we would like to do with the workflow template that is generated before it is deployed. For example, adding custom metrics, adding title/description, or customizing ttl. I guess that these (etc) could be added a features directly to Metaflow, but I'm wondering whether there is (or could be) an escape hatch for generic 'postprocessing' would allow us to customise the workflow output without having to have them added to the Metaflow spec.
āœ… 1
Some ideas: ā€¢ add a lifecycle hook for flow decorators that could modify the compiled workflow template before it's applied ā€¢ add a separate command 
flow.py argo-workflows compile -o ./workflow.yaml
which does everything up to and including the compilation step, but outputs it to said file. Then update 
argo-workflows create
to accept an optional precompiled workflow. Advantage of the second is that it'd additionally fit a gitops style workflow, where configs are committed to git and applied separately (e.g. via ArgoCD)
m
At my company we are using the Metaflow extensions framework to get this type of functionality. We so have adapted the Metaflow CLI for Argo to generate the vanilla argo workflow manifest, we then update it and then either (i) write it to file as part of our gitops deployment process or (ii) submit it to the cluster for remote development.
I was actually going to discuss what we have done in an upcoming Metaflow office hour session on the 26th of Sept.
c
Nice! Glad to hear that there are others with the same use case. Could you clarify what mechanism you're using as part of the 'extensions framework'? And when you say you have adapted the CLI, you mean that you are maintaining a fork that adds that functionality?
m
https://github.com/Netflix/metaflow-nflx-extensions - So it is (unsupported) functionality within the Metaflow codebase. It allows you to do a bunch of stuff, like add your own decorators or overwrite existing CLIs.
In our case we overwrote the Argo Workflow CLI and then also added a flow decorator that allows us to add flow level configuration.
c
Ah, we use this pattern to add decorators, but I had failed to realise it allows overriding existing CLI methods.
m
ah, cool. Yeah, you can do CLIs too. šŸ™‚
šŸ„³ 1
5 Views
Open in Slack
PreviousNext
Powered by