Hi all, Is it possible to have Metaflow integrate with custom backend orchestration systems? At the company where I work, we are using an in-house built orchestration engine (which is written on top of Argo and runs on GKE and GCS). What do I need to do to have Metaflow integrate with our system? Thanks

sure. It’s not necessarily easy but you can definitely write your own custom backend. At Netflix we use Maestro and have our own implementation. If yours is close to Argo you can probably reuse a lot of the code from the argo integration. You would need to develop a Metaflow Extension (https://github.com/Netflix/metaflow-extensions-template). Happy to help if you need anything specific. It should allow you to create something like a

myorchestrator

backend and you would then be able to do

python ./myflow.py myorchestrator create

for example.

also, here are a few integrations to derive inspiration from - 1. argo-workflows 2. step-functions 3. airflow