The human-centric platform for production ML & AI

Outerbounds

I'm just starting to use Metaflow and have realized that it would be great for building small, self-contained data pipelines.

But, how can I create one big pipeline from all of my little pipelines?

Is it up to the end user to build their own orchestration system for different flows, or is this something that's already built-in?

It looks like you can test flows of flows locally with @trigger_on_finish(flow='FlowName') and you can declare flows as dependencies of other flows in Python.

It looks like you can deploy flows that depend on other flows to Argo, can you do the same thing for AWS Step Functions and Apache Airflow DAGs?

currently the trigger_on_finish is not supported with Stepfunctions or Airflow. Its only supported with Argo.

Gotcha, so if I'm deploying to AWS, I'd need to write my own orchestration logic in AWS EventBridge?

if you want to only use event bridge you will have to setup something custom or bake it into Metaflow's step-functions integration. You can use argo events if you end up deploying EKS on AWS :slightly_smiling_face: