I'm looking for feedback on how others have handled defining scheduled Argo workflows as code. I'm the the infrastructure lead for a team of ~20 data scientists, so I'm hoping for a solution that will allow the scientists to schedule workflows with minimal complexity. Here are our two current trains of thought: • Create a pipeline that authenticates with the kubernetes cluster so that we can run things like

python example_flow.py --with retry argo-workflows create

and have the pipeline automatically deploy that command • Run the metaflow command locally, and copy/paste the argo yaml spec to the repo. Use a argo API key + CICD to deploy the spec (see here) I suspect the second option is the more flexible one, because then we can modify the yaml spec to do things like post alerts when jobs fail

@elegant-beach-10818 - while both the options work, it will be easier to use the first option (especially going forward where Metaflow will instrument event sensors on your behalf)

Is there currently an escape hatch to add arbitrary configuration to the argo workflow spec? Adding things like event sensors would have to be manually added via the argo UI with the first option, and then if the code gets updated they would have to be manually added again

The trouble with introducing arbitrary configuration is that it may conflict with the functioning of Metaflow. If you have specific things in mind, I can definitely help with a strategy to add those in.

My main goal is to output notifications of failed flows to slack

I am looking into adding that as a first class feature in our integration

Awesome thanks! In the mean time if team needs that I should be able to manually add it to the deployed argo yaml template

yes!