Hi All I was wondering whether it would be possible to split out the

_compile

function in the

ArgoWorkflows

class a little bit? I've made a Github issue here. Think it would be a very small PR and would be backwards compatible. For context, we are writing a Metaflow extension for the Argo Workflows functionality. One of the reasons for doing this is so that we can add our own configuration to the

WorkflowSpec

for the workflows, e.g., so that we can add exit hooks. The idea of the proposed change is to make it easier to subclass the existing class. Would be great to get some feedback. Thanks. 🙂

@mammoth-rainbow-82717 are you open to a quick call to discuss alternatives for introducing exit hooks? The ‘_compile’ method is an implementation detail and depending on it for your implementation of exit hooks can turn out to be quite brittle.

We already use exit extensively with our deployment of Argo Workflows, so I think we'd definitely like to keep them. Not least, I think it would probably make sense to keep out Metaflow flows consistent with our vanilla Argo pipelines on our cluster. Why do you think they would be brittle?

As we introduce more functionality within the Argo ecosystem, our implementation of the integration is likely to evolve and directly depending on private methods will couple your extensions to a specific version of Metaflow. That will make upgrading patch versions of Metaflow a bit tenuous for your team.

I see. We are kind of signing up to that when using the extension approach anyway. It is something we have already discussed internally and is something that we are aware of.

Have you explored using Kyverno to intercept and modify the Argo Workflow Template on Kubernetes?

We don't currently have a policy agent available on the cluster. Though, it is something we might look into in due course. Would be out of scope for our initial deployment of Metaflow though, I think.

There are a few different ways to do that. You can set a custom annotation (that we can introduce support for) that the policy agent looks at and adds the necessary hooks. Yet another alternative is to have a simple script that takes the output of

python flow.py argo-workflows create --only-json

and adds the relevant bits of your customization into the generated json before submitting it to Kubernetes.

The second approach is similar what we are doing. We are actually writing our own cli through an extension, but essentially it is very similar. I guess my proposed approach is just not quite the correct approach. I guess I could generate the WorkflowTemplate and then edit it, like you say.