Hi folks, I realize this question might be difficult to answer without more context but I’m looking for broad strokes… How do people generally organize their flows? Say you have one process performing some kind of ETL, another that is enhancing the artifact of that ETL with some meta-analysis, and another that’s doing post-processing on some prior artifacts. Do you try to represent them all in one flow? Do you have several flows that are configured independently and then an over-arching “master” flow that can coordinate the whole thing when needed? If anyone has any insight, examples, or resources I could dig into I appreciate it. Thanks

I'd love to hear from other folks too, but I can quickly comment that it is very typical to have multiple flows. For instance, one flow to train a model and another flow that uses it to produce fresh predictions.

Interesting. Thanks @straight-shampoo-11124

If it helps, I can share my experience as well 🙂 At our organization we use metaflow primarily as ETL orchestrator rather than for ML, because well it’s convenient. We deploy flows as step functions and we have an in-house gRPC service written in python that provides a cross-language programmatic interface to trigger flow runs (via SFN api), as well as monitor running/past flow runs (a combination of SFN & metaflow API). Once the data scientists get their run id, they can use the metaflow api to do their analysis if needed. We use this interface to create chains of multiple flows if needed. The cross-language part is our specific requirement and may or may not apply to everyone, essentially we have some backend code written in other languages that needs to be able to call a number of flows and wait/poll for them. In future we are also looking at adding some in-house support for subflows as well as that has been a long-known wish of our data scientists. I know SFN (our target for production) has first-class support for sync/async triggering other step functions which we intend to leverage for that. This is really invasive though and would require number of changes to metaflow core codebase so we’ve beein holding it off so far 😛

Thank you @quick-lighter-52296!!