Has anyone "linked" metaflow with azkaban python client? I'd like to do a quick POC for my group.

Dont think so.. A mere grep about 'Azkaban' brings no results, but a few records in a sample dataset file, about the JK Rowling work that helped coin that Once-upon-a-time-great scheduling/orchestrating tool.. 🙂 @ancient-application-36103 @victorious-lawyer-58417 I'm not sure if orchs like Azkaban, Luigi, Dagster may have to be also supported, but FWIW it should be a low-hanging fruit to implement since Azkaban AFAIK is all about "packaging" jobs as a zip and has had well defined APIs(over CLI and REST).. I can try a shot at writing integrations of Metaflow to Azkaban and possibly Marc/others can collab if it helps the community!

right, no Azkaban support yet. In general there are two ways to support another orchestrator: 1. You can run

python myflow.py run

as a normal Python step with any orchestrator. 2. You can translate the Metaflow DAG to the DAG of another orchestrator. The first approach can be very straightforward and easy to implement. The second approach can be really hard, since many orchestrators don't support all features that Metaflow needs (e.g. Airflow doesn't support arbitrary subdags in foreaches).

@victorious-lawyer-58417 @cold-elephant-44576 re: the 2 ways: 1. hmm. basically in any orchestrator on whatsoever mode/layer a node of a DAG/workflow gets to run the metaflow flow upon, be it bare OS, a virtualenv, a container, a pod etc. we can just execute the flow with aforementioned command as if we are running it "locally". This is more of a hack/workaround, but yes, effective (esp. for trivial, not so complicated flow/dags). 2. While we certainly need a "translator" as integration/support for multiple operators, it may not be as hard, as for example, with Airflow we actually have "Dynamic Task Mapping" which is akin to the foreaches (IMO still just a task and not a "subdag" but then Airflow also has "Task Groups", >=2.6), please see screenshot below.

yep, the current Airflow integration supports such basic foreaches, which is a subset of foreaches that Metaflow supports. Implementing all details of this is a bit non-trivial as you can see here.

Thanks for all the answers. I was thinking about approach 1 to be frank. This is very far from perfect since you lose some very handy things like '@gpu' decorators and you concretely might have to do it multiple times as you might need to run your dag on multiple environment (ie. Etl with a spark container and model building with a GPU instance). Therefore "connecting" theses 2 tasks (which would only be 1 task if you didn't have to simulates it is local) might not be possible. That being said if you have any examples this could be helpful as it can be a first step for us for a POC.

this repo has an example of approach 1 using Prefect