Super basic question around documentation concepts on https://docs.metaflow.org/getting-started/infrastructure#supported-infrastructure-components. Would it make sense to think of the Deployment and Orchestration layers as the same? But with 4 options: • local • argo • step functions • airflow Or is it rather that the "orchestration" layer only has the local option in some sense?

you can consider Orchestration as a subset of Deployment. Many deployment patterns benefit from a robust orchestrator, which is why Metaflow makes it particularly easy to move from local orchestration to production orchestration. Deployed flows have additional operational concerns, e.g. how to integrate the flows to surrounding systems, for instance through event triggering. Also typically you want to make production deployments more fault-tolerant. Deploying models as a microservice (aka model hosting/serving) would be another deployment concern. While Metaflow doesn't have a built-in model hosting solution today, many companies deploy models from production orchestrator to some model serving solution like Sagemaker.

thank you!

It would be great to add an integration with dagster (similar to how it's done for meltano) to aid orchestration of external tasks (meltano pipelines for example) which are out of Metaflow scope and make life easier (vis-a-vis step functions) in general: https://dagster.io/blog/dagster-meltano-integration-tutorial

good feedback, noted! for context, if you wanted to use Meltano with Metaflow today, there are a few options: 1. You can execute

meltano run

in a Metaflow step. This is what the Dagster integration seems to do under the hood as well. 2. You can use an orchestrator that's natively supported by both Metaflow and Meltano - Airflow should work at least. 3. You can use a separate orchestrator for Meltano (e.g. Dagster) and Metaflow (e.g. Argo Workflows) and then use even-triggering to connect the two.

Thank you very much Ville. Yes, airflow is an option, although it takes a bit more time to debug (also meltano-airflow utility lacks functionality and is not customisable out of the box). In my case I'm using built-in meltano extractors and loaders and dbt utility to get properly cleaned dataset, but it still needs some additional sparking) Dagster has decent support for running spark jobs on databrics or emr cluster. Metaflow supports serverless spark jobs, but I'm yet to figure out how to debug those, so sticking to emr for now. Will explore event-triggering later.