Hello! I am taking a look at running metaflow on eks w/ airflow. I'm coming at this from the infra side and @aloof-application-80784 is bringing the data science. We are both only slightly familiar with Airflow and were wondering if you had example dags or guidance to take advantage of the architecture shown in the eks_airflow example?

Hey Will 👋 , We have not made the official release of running Metaflow on Airflow but many parts of the code have been merged into the metaflow code package. Since the we have not made the official release the API is not documented on the website 😅 . If you are interested in using Metaflow on Airflow, you can start with the docs over here. This is the running google doc that holds all the information about running Metaflow on Airflow with Kubernetes. Currently the public metaflow package consists of an airflow integration that *doesn't support Foreach's*; We will be supporting foreach's which are compatible with Airflow's style of dynamic task mapping very soon. Please reach out if you need any more assistance.

That makes sense, I thought I was missing something 😅 We will take a look, thank you!

Why airflow? If you are on eks already, I am having trouble seeing any advantage over Argo.

Our recommendation is to use Argo Workflows on Kubernetes. But if an organization is already using Airflow on top of Kubernetes, we want to make it easy for them to adopt Metaflow without committing to a new orchestrator just yet. One benefit of this approach is that any future migration from Airflow to Argo Workflows will not affect any existing Metaflow flows.

Oh I get your and outerbounds motivation @square-wire-39606, although I don’t envy having to support it. :-) I was curious as to @wide-battery-59962’s motivation as he said in the original post they had little experience with Airflow.

Same as what savin said. Larger organizational mandate for consistent tooling.