Hey team! It seems like multiple linear steps after a foreach are not currently supported by Airflow-orchestrated Flows:

Not yet supported with Airflow:
The foreach step format_dtype created by step load_datasets needs to have an immediate join step. Step impute_null is invalid since it is a linear step with a foreach. This type of graph is currently not supported with Airflow.

Is support for this type of graph and/or nested foreaches on the roadmap for the Metaflow team? My team is looking to begin deploying/scheduling our Flows, and we want to understand if this capability is being prioritized/worked on in the Airflow integration, or if we should consider using Argo Workflows/Step Functions.

@green-analyst-32514 unfortunately that is a limitation of Airflow - once it's lifted on their end, we can look into introducing support within Metaflow. Currently, Step Functions and Argo Workflows support such DAG structures.

^ My team has run into this as well. Our Data Engineers use Airflow and wanted the Data Science team to as well. However, we have several Flows that nest foreach loops. We’re targeting Step Functions for deployment now. You can try asking on the Airflow Slack to see whether or not this is something they plan to tackle

Gotcha- thanks for the responses! 🙏

Hi all, I’m curious how the nested Airflow support turned out?