Are there any documents or tutorials on writing a new plugin for metaflow?

Hey Clif ... 👋 The Metaflow codebase has a few plugins and you could look at one of them to see how it works (e.g. Argo plugin). However, before getting into the details of it, it'll good to discuss what are you trying to do. Let's follow-up offline and I can share a summary of the conversation here for everyone. WDYT?

There is this as well: https://github.com/Netflix/metaflow-nflx-extensions

in general there are many ways to extend Metaflow, depending on your needs. Here are the main ways, starting from the simplest to more advanced approaches: 1. You can write a custom decorator as a normal Python decorator which can manipulate artifacts and environments before and after the user code executes (e.g. see this `@skip` example or another example, `@resumable_processing` ) 2. You can make a custom card that executes arbitrary code after a task has completed and produces a visualization. 3. You can use Metaflow's internal extension mechanism, which @dry-beach-38304 linked above, to create extensions that introduce new commands and decorators etc. A downside is that the API is subject to change and it is a more complex beast.

Here's a quick summary: • Armada is an OSS project which provides ability to run workloads across multiple K8s clusters. • Teams at G-Research are the maintainers and main contributors to Armada. • They want to build an integration of Metaflow with Armada. i.e. users should be able to write Metaflow flows and run them on Armada • Armada already has a python client AND an Airflow integration. So the expectation is that providing metaflow integration should be easy. Technical details • Armada presents it's own REST API endpoint. Workloads are submitted to this endpoint. Armada will them send this workload to one of it's backend K8s clusters. The workload itself gets an ID for subsequent operations on it. • There is an API (

binoculors?

) with which, given an API, it can return the status of the job, logs, etc. This API will get used. User exeperience: There were two tracks of discussions about this. 1. Providing a CLI similar to how argo, airflow integrations are. E.g.

python flow.py armada create

OR 2. Providing an

@armada

decorator that can submit one step to Armada Similar to argo/airflow, there is a need to construct an Armada specific job object here and submit it to a specific endpoint. That suggests #1 above would be the way to go. But, it will have to reside in the main code base. Is that correct? However, similar to

@kubernetes

, one can imagine, simply running only specific steps in Armada. And therefore even an

@armada

decorator seems plausible. The benefit of it's own decorator is that it could be more maintainable and reside as it's own independent project. @rhythmic-gpu-85801 has a small demo of option #1 above. @victorious-lawyer-58417 / @ancient-application-36103 / @dry-beach-38304: Would be great to get your thoughts on this before going too deep in the implementation.

very quick comments (and I haven’t spent a huge amount of time digging so take this with a grain of salt): • your two use cases definitely fall in 3. (the extension mechanism). Both are totally do-able. Specifically, no, nothing of this would need to live in the main OSS codebase and you could have a metaflow armada extension that could provide either or both of your use cases. • there is a key difference between airflow and kubernetes (or said another way between your 1. and 2.): ◦ metaflow -> airflow converts the DAG into an airflow graph (or whatever their nomenclature is). airflow then runs the tasks on underlying hardware (a k8s cluster for example). When executing, it’s airflow’s runtime/orchestrator that schedules things. Metaflow is at that point just used as a library in each of the tasks running. ◦ metaflow w/ kubernetes decorator: metaflow acts as both a runtime and a library. It sends to kubernetes a single task (not a graph) to execute. It replaces the airflow orchestrator in this case. Said another way, the first integration is a scheduler integration, the second is a compute integration. Both are definitely possible and it sounds that if armada can also directly interact with airflow, that may be an option (ie: you convert first to armada which can then convert to airflow) but I am unclear of the benefit (maybe the multicluster thing?). If armada can also act as a simple front to a compute resource (the k8s cluster), option 2 also makes sense.

No worries at all .. this is great! And you'll have a lot more insight into how to go about navigating these options. I'm learning..

excited @User @rhythmic-gpu-85801 shall we do a quick chat to figure out the way forward? Excited to see the demo too!

@rhythmic-gpu-85801: Let us know how this effort is proceeding and if there are any more questions. Would be great if we want to sync up again and see a demo.

I'm making progress. No blockers ATM. Hopefully able to demo something more substantial in the not too distant future.