Hi I have a question regarding argo workflows plugin, i am trying to implement our own

ArgoClient

to make metaflow call Rest API requests instead of k8s API when talking with argo, wondering what's the preferred way of doing it while using the

metaflow_extension

library. And how to deal with future updates in that case?

you can create your own subcommand using the extension functionality - just a word of warning that the extension API is not backwards compatible yet so technically it may change over time (although changes are not frequent)

No particular advice just off the bat but happy to answer questions or review anything you want me to look at. We do have an extension for our own orchestrator within Netflix; it follows a similar pattern to thr one in OSS, namely converting the graph thst metaflow has to one that the orchestrator supports. As @victorious-lawyer-58417 mentioned, we do not guarantee that internal APIs or things extensions rely on will remain stable but historically changes are very rare. Finally, I will add that with thr extension you can also override stuff (so you don’t have to change the name and can use the same name command as in OSs except do something different.

right, then if for example if i just need to override

ArgoClient

in

argo_client.py

, do i need to have everything under

plugins/argo

in my

metaflow_extension

lib? right now i need to have my own version of all the files here in

metaflow_extension

for it to work, is it possible to only override one file while keeping other files as per what original metaflow provides?

I have not tried this yet but you can try the following: • have your

argo_client.py

file in

plugins/argo

• set

__mf_promote_submodules__ = ["argo.argo_client"]

in the

mfextinit_myextension.py

located in

plugins/

It may work. It’s going to depend on the order of loads but there is a chance it works. What this does is point

metaflow.plugins.argo.argo_client

to

<http://metaflow_extensions.my|metaflow_extensions.my>_extension.plugins.argo.argo_client

. I saw there was only one import from argo_client in the code and it is through

from .argo_client import ArgoClient

so it’s possible this works. If it doesn’t, let me know and I can take a closer look. I haven’t tried overloading a single file like that quite yet but in principle it should work.

Hey @brief-kite-90012, any chance you can share your argo implementation that uses the REST api as opposed to the K8 implementation?

Hi @stale-eve-11739, what I did is just using the metaflow_extension template and overriding

metaflow.plugins.argo.argo_client.py

with rest api request calls instead of k8s api for each of the methods, I'm using it with

metaflow==2.12.6

, you might need to tweak

argo_workflows.py

as well but I am not sure.

thanks @brief-kite-90012! will try it out this is probably useful for a lot of people so might be worth pushing this out as an independent plugin