Questions about composability We are looking into adopting metaflow and have a couple of thoughts related to composability This question to us feels as if on some level it may have various parts to it depending on the context. In particular we are thinking of basically Metaflow as two sorts of things. Inside a step I'd say you basically have "real python" where everything should just work. These steps are then tracked and orchestrated using a python syntax but in reality, this is much more than python. It's really more accurately S3, yaml, docker, argo, kubernetes (and a couple of other pieces) all glued together to be able to be presented to the developer as merely python. Really it's a stack in a python trenchcoat if you will. And this is very elegant and an admirable job has been done of making this feel very easy to the degree that it took us while to quite appreciate where this boundary lay and how much work to put on either side of this boundary. If anyone with more experience wants to correct this mental model that would be helpful. That said it leads the question around composability to essentially be multi pronged: 1. In this upper realm above the step boundary it seems as if there are tools for composing flows via argo triggers in the cluster but there are limitations in terms of the development cycle leading to a feeling of needing to copy paste steps between flows potentially especially while iterating on flows while creating them to begin with. What would be nice at least theoretically might be the ability to act on flows more directly in the code prior to deployment. By way of example it might be nice to be able to compose flows by addition like so:

BigFlow = FlowA + FlowB

2. Along a similar line of thinking it may be desirable to be able to inline another flow inside of a flow definition maybe using something like an

@InlineFlow

decorator that then places steps already defined as another flow into that location in a flow allowing incremental development of subparts of a larger flow. 3. Finally on the issue of composability down into the steps this is where the boundary I described above becomes more difficult to deal with. I want to emphasize I don't dislike the existence of this boundary as it allows clarity in terms of keeping track of what code runs where and consequently improves the ability to reason about data locality, vertical scaling optimizations etc rather than trusting some general purpose scheduler to get it right. Here i guess the inquiry would be around the idea that the units of work intended for a step may actually be fairly large and complex. The existing flows and steps api's are very nice to work with and just beg to be used inside these step boundaries to arrange work. Obviously, the parallel map exists and we have actually seen great performance in our tests when using dask inside a step to parallelize work as well. I'd imagine there are one or two other packages in the python ecosystem that may also work well here. The desire here then would be to extend a possibly more restricted syntax very similar to that of the steps themselves down into this context possibly backed by parallel map or dask or ray some other such engine. This seems like it would be a convenient and self similar sugar that could make it easier to chain, split and merge work steps even inside of the steps. Maybe they could be called sub steps or something like this Unlike many in the community that have likely actually built complex systems using metaflow these questions come entirely from our explorations of the stack as we consider adopting it so they may reflect misunderstandings and just outright skill issues so if that's the case I would appreciate corrections of the implied mental model that more experienced members can see underneath this line of questioning. But of course direct comments on these ideas would also be interesting. Thanks ahead of time.

There’s a @trigger_on_finish(upstream_flow: str), flow level decorator that you can use to chain flows together. It’s a pub sub, where downstream flows subscribe to completion events of upstream flows.

I really appreciate the thorough reply @lively-lunch-9285. I think the concern is not so much about running them as the development lifecycle and code reuse. Yes you can chain them serially in infra but that isn't the sort of chaining I'm really talking about. I think what I'm talking about is the idea of being able to have different developers working on different parts of a complex flow that will eventually be deployed as a single flow and having the ability to build up parts of that flow as flows that can then be composed together ultimately for deployment in prod to run all together but at development time can be run independently. Now you might argue that exactly what triggers are but i think the main shortcoming of that is they can't easily be used locally and to me the ability to do the same thing locally as in prod is one of the most appealing features of Metaflow. Composability of flows directly in python code would help to complete this capability in my mind. I do appreciate the referral to Hamilton. ill check it out. that said I guess the reason I brought this up wasn't because of a lack of tooling. As I mentioned we have used dask for exactly this idea to great success, it was more intended in regard to enjoying the metaflow flow api and preferring that this be extended to maybe wrap some local dag runner like dask or hamilton in an integrated manner that maybe one could inspect from the metaflow ui etc Hopefully that clarifies the intent and scope of the discussion a bit.

Oh gotcha.

triggers can’t be run locally

Actually, I think Metaflow recently launched a way to run FULL Metaflow locally with a CLI called

metaflow-dev

which runs minikube https://docs.metaflow.org/getting-started/devstack

Yeah we have been using the dev stack quite a bit as we don't yet have a production instance. as you say its not difficult to run necessarily but it really adds complexity to that first tier of development effort and essentially force the use of dev stack or a cloud instance somewhere which feels more limited than just directly being able to run everything in python and have this gradual step up in complexity and consequently development cycle times. I'm especially interested in the second point as that seems to get to the core of my concern. in a flow the interface is very nice but then the interface seems to break down and you cant really compose easily beyond that. If there is work a foot to attempt to better the interface to allow step reuse that would already go a long way towards fixing my code reuse and parallel development concerns. Do you happen to have a link to some artifact around this? maybe github issue or something?

re: testing triggers - you can actually test them locally - 1. @trigger_on_finish - https://docs.metaflow.org/production/event-triggering/flow-events#testing-flow-triggering-locally 2. @trigger - event payload gets mapped to parameters - so it's equivalent to running the flow locally

conditionals and recursions are now available