Hello there! First, thank you for building and maintaining Metaflow. I really appreciate the clean syntax and the seamless transitioning from dev to dev-at-scale to prod-at-scale approach using K8s and the Argo stack. Second, I am looking for information or help, as you see fit. I am building a pipeline, that I want modular. The desired architecture for this pipeline would consist of a hierarchical decomposition where level-0 would be a Metaflow flow, calling level-1 objects being Metaflow flows too. Level-1 could either execute steps, or itself call level-2 objects as Metaflow flows. The result would be nested workflows. And the rationale is that each object represents a semantic module, that could be substituted by a variant as needed, hence the modularity of the pipeline. I have a good idea of how to drill down the nested pipelines using events triggers. ArgoEvent().safe_publish() being used to trigger level-1 from level-0, and trigger_on_finish() being used to chain two level-1 objects as needed. And so on... Where I struggle is when it comes to monotoring progress from the upper level flows. Ideally, in flow level-0 the first step would trigger one or more level-1 flows. And then the second step from level-0 would start only when the first step is complete. This suggests some kind of bidirectional communication between parent/child flows, e.g. parent flow listen to an event that child flow is done, then pulls its outputs through the Metaflow API. Is anyone aware of similar examples of such nested workflows with back and forth communication? I am not settled on any architecture to achieve this, and will gladly take any suggestion from the community 😁 Is it possible to listen to an ArgoEvent from within a flow? (E.g. to wait on an event) This is more oriented towards one path forward that I see to implement "Workflow of workflows" design pattern. I hope my case will elicit some interest. Anyway, please keep on developing Metaflow, this is a neat project!

meow wave this sounds super cool! to monitor the progress, you can use the metaflow client to check if all the child flows are successful within the first step before moving to the next step in the parent flow.

Thanks for the suggestion! Going through the Client API is definitely a good option and I will explore that soon 🙂 One thing that is needed for this approach is that the parent Flow (or rather the Run) knows the name of the child Run. Only this enables monitoring through the Client API, as far as I understand. Otherwise it is hard to guess which Run to monitor among all the Runs going in parallel for the child Flow that was started. And it is still unclear to me how to get this information programmatically. I would know how to do that by submitting a workflow from Argo CLI, but not from Metaflow directly. Maybe I missed this information from the doc, would you have a suggestion about how to achieve this?

you can also know this information by submitting through metaflow -

python flow.py argo-workflows trigger --run-id-file foo

Looking into this, and I will share progress (or further questions) 😁

Great question! @brainy-truck-72938 is implementing this specific feature right now

I'd love to know more! @brainy-truck-72938 , would there be a place where this code is publicly available already?

Currently being done at https://github.com/Netflix/metaflow/pull/1732 Only the last 2 comments should be relevant now.

Thanks for sharing, I'll look into it!

There are basically 2 levels • a low level chaining API that spits out a command for us which we can then pass onto a subprocess call • a higher level API which should be much cleaner — this is what will be exposed to the end user. The initial plan is to start with the Runner API and then slowly cover other things such as argo-workflows…