Hi there. We use Metaflow and it works great for spinning up hundreds of machines to prepare big datasets. One flow goes through a series of steps and scales out horizontally to produce one dataset or one model. What I think we are lacking is to effectively tie these flows together in dependency graphs like you would with Luigi or DVC (automatically computing the missing artifacts in your dependency graph). Does anyone have experience combining pipelining tools with Metaflow?

@thankful-father-61351 is the intention to create a DAG of DAGs?

@square-wire-39606 The intention is a DAG of artifact-dependencies like you know it from other pipelining libraries. We could do this with Metaflow, but it would require a lot of custom code, and I would prefer not to build it myself.

Gotcha - we are discussing this internally. I will follow up as soon as I have more details

The more I think about it, the more I end up wishing that I had an artifact-first orchestrator, where everything was about artifacts. In the utopian future scenario, that would allow me to have an artifact registry and even to be able to “rebuild” artifacts, since code-packages and everything is tracked. I have the hunch that it will be where at least some part of the MLOps-world will be moving towards. But this would be a very fundamental change to Metaflow that of course should be met with a healthy amount of scepticism 🙂 Just because I need something does not mean that the rest of the world needs it. Just sharing my personal outlook.