Hello there! First, thank you for building and mai...
# ask-metaflow
e
Hello there! First, thank you for building and maintaining Metaflow. I really appreciate the clean syntax and the seamless transitioning from dev to dev-at-scale to prod-at-scale approach using K8s and the Argo stack. Second, I am looking for information or help, as you see fit. I am building a pipeline, that I want modular. The desired architecture for this pipeline would consist of a hierarchical decomposition where level-0 would be a Metaflow flow, calling level-1 objects being Metaflow flows too. Level-1 could either execute steps, or itself call level-2 objects as Metaflow flows. The result would be nested workflows. And the rationale is that each object represents a semantic module, that could be substituted by a variant as needed, hence the modularity of the pipeline. I have a good idea of how to drill down the nested pipelines using events triggers. ArgoEvent().safe_publish() being used to trigger level-1 from level-0, and trigger_on_finish() being used to chain two level-1 objects as needed. And so on... Where I struggle is when it comes to monotoring progress from the upper level flows. Ideally, in flow level-0 the first step would trigger one or more level-1 flows. And then the second step from level-0 would start only when the first step is complete. This suggests some kind of bidirectional communication between parent/child flows, e.g. parent flow listen to an event that child flow is done, then pulls its outputs through the Metaflow API. Is anyone aware of similar examples of such nested workflows with back and forth communication? I am not settled on any architecture to achieve this, and will gladly take any suggestion from the community 😁 Is it possible to listen to an ArgoEvent from within a flow? (E.g. to wait on an event) This is more oriented towards one path forward that I see to implement "Workflow of workflows" design pattern. I hope my case will elicit some interest. Anyway, please keep on developing Metaflow, this is a neat project!
1