Is there anywhere a comparison between using Argo ...
# ask-metaflows
stale-florist-77883
06/03/2024, 6:04 PMIs there anywhere a comparison between using Argo Workflows and Apache Airflow orchestrator when deploying Metaflow to Kubernetes? From what I've read in the docs, it seems the only difference is that triggering flows from within Metaflow is possible in Argo Workflows and not in Apache Airflow.
Which one would you recommend and why? (Considering it will be used only with Metaflow)
s
square-wire-39606
06/05/2024, 2:06 AM@stale-florist-77883 the deployed flows on Airflow and Argo Workflows behave as native Airflow DAGs and Argo Workflows WorkflowTemplates. If you have access to an API that can trigger Airflow DAGs, you can easily invoke that API from within Metaflow.
square-wire-39606
06/05/2024, 2:09 AMre: using Argo Workflows vs Airflow - Airflow doesn't provide supported for arbitrarily nested foreaches - so that would be one limitation. Ideally Argo Workflows is a better solution if you are looking for running a large number of flows or very large flows (we are able to support flows which launch 10k-100k containers easily). With Airflow you are limited to a much smaller number unless you resort to optimizing your Airflow setup. None of these limitations are enforced by Metaflow itself.
square-wire-39606
06/05/2024, 2:09 AMArgo Workflows, being k8s native, allows us to offer reactive triggering which is currently not available in Airflow
s
stale-florist-77883
06/05/2024, 1:06 PMThank you for the clear description! 🙂
3 Views