I’ve updated it since the discussions above, so hoping to move most discussion to the draft PR (and smaller PRs that I will factor out of it)

Also happy to discuss it further here.

there are great ideas in this PR. Some ideas like flow composition and better support for

pytest

are definitely something we want to support in the future. Other ideas like getting rid of

self.next

are more controversial and they would break all existing flows or add complexity, if both styles were supported. Hence it is hard to consider merging this PR as is

we could cherry pick some diffs from the PR over time

Check out test_simple_foreach.py to see one example of an old-style and new-style flow defined side-by-side, and a test that runs each one and asserts their data artifacts

lmk if this addresses some of your concerns, and I’ll update the PR description to make some of this more clear.

I’ll update this thread soon re: the next steps

Noah and I are on ET, I’m generally pretty free, feel free to suggest some times that work for you.

We could even record it, maybe some good documentation will come out of it

@creamy-magician-26843 is there a specific part of this PR (it contains many changes) that is of interest to you?

I don’t think we recorded the meeting; some of my belated notes/takeaways: • @straight-shampoo-11124 is open to

pytest

tests existing alongside existing

test/core

tests •

metaflow flow …

CLI seems like a good thing overall; some concern about letting there be 2 ways to invoke flows (

python <file>

still works, when

__main__

handler is present, as is always the case today). Will that confuse users? • Need to drill into which bits of #612 actually depend on one another; right now a lot of things are separate but in sequence, though some of them could be completely factored out and not require any of the other bits to go in first. I’ve filed #661, #662, #663, #665, and #666 today, to start that process. Composing Flows As currently written in #612, inheritance only works with “new-style” `metaflow.api.FlowSpec`s that omit

start

and

end

steps, because I punted on trying to handle steps with the same name being mixed in from multiple flows. An open question is whether/how to namespace steps (in a composed flow, after composition) based on the flow they came from. For example:

class Flow1(FlowSpec):
  @step
  def start(self): self.next(self.end)
  @step
  def end(self): pass

class Flow2(FlowSpec):
  @step
  def start(self): self.next(self.end)
  @step
  def end(self): pass

class Flow12(Flow1, Flow2): pass

What should the 4 steps in

Flow12

be named? •

Flow1_start

, …,

Flow2_end

(underscore-delimited): probably easiest to do, but a bit hacky/brittle; steps can have underscores in their names already, so it’s not a robust way to namespace steps. •

Flow1/start

, …,

Flow2/end

(slash-delimited): seems cleanest, but might interact undesirably with existing assumptions around metastore directory structure. Or, perhaps it will Just Work, and introduce a new level of directory hierarchy that is never ambiguous? •

Flow1.start

, …,

Flow2.end

(dot-delimited): might give the best of all worlds, but might also break assumptions about step names. • or, maybe `Flow12`'s steps should actually appear to Metaflow as just {

Flow1

,

Flow2

} x {

start

,

end

}, and the

Flow12

metadata should just point to invocations of those `Flow1`/`Flow2` steps, somehow?

Metaclasses, Python 2 In any case, my current approach (and most other versions of it I can imagine) rely on metaclasses (that I added to both new-style and existing `FlowSpec`s), which give a useful hook for computing things about flows at declaration time (as opposed to at run-time, as happens today). However, I think using metaclasses requires dropping Python 2 support, which introduces a potentially big blocker for this work. I guess I saw a Python2 metaclass shim in the code-base, but I can’t justify spending time making things work on Python2. I filed #668 about dropping Python 2, and #662 which drops it from the CI. Perhaps we can start a

3.x

branch and I can focus on landing things there 🤷. Next Steps In general, @straight-shampoo-11124 felt he and the other maintainers have some homework to do in thinking through the large possible design space around composing flows. There’s also a fair amount of stuff in #612 that can go in first (either because later stuff depends on it, so it has to go in first, or because it’s orthogonal, and can go in separately whenever). As mentioned above, I’ve started filing PRs with some of that stuff: #661, #662, #663, #665, #666. lmk if I missed or misrepresented anything @straight-shampoo-11124, and as always I’m interested in folks’ thoughts about these topics more generally. Perhaps we can move them to more focused GitHub issues as well.

(jfyi: slack broke up my message above and only put the first ≈half into the main channel, for those not following the thread directly)