https://outerbounds.com/ logo
Join Slack
Powered by
This is an ad I just saw for a Metaflow competitor...
# ask-metaflow
l
This is an ad I just saw for a Metaflow competitor called Union.ai . Thought it could spur discussion. It took me 4 times, but I think I've spotted a trend 🤣. Could there be an advantage to Metaflow adopting a similar interface? I've noticed a lot of new pipeline tools are using this abstraction and I kind of like it. Rather than wrapping everything in a class, you decorate functions and chain them together in a controller function decorated with 
@pipeline
(ClearML and ZenML) 
@workflow
(Union ai), 
@graph
(Dagster) etc. Possible advantages: • fewer classes (mostly joking), • I guess you could technically share steps between DAGs since they are simply importable functions. Hard to import a single step out of your flow if it's embedded into a class. • kind of feels more intuitive. I tend to reason about a DAG as a series of steps. In a sense, it's weird to have that sequence of steps contained in a class.
f
can also add prefect to that list w their 
@flow
decorator: https://docs.prefect.io/2.14.3/concepts/flows/
l
addition-to-mycollection-more-collection.gif
f
there we go 🙂 Interesting union has pandera on their list of products - may be misremembering but IIRC that project was a solo-dev effort. Makes sense in a product like that
l
I've heard of pandera, but haven't used it. It's cool it got picked up by something bigger.
Oh duh, I missed one of the biggest potential advantages/disadvantages: the function-based interface seems to encourage you to use python control flow logic to define which steps run in parallel, (or conditionally), and over which data. Is Metaflow's interface limiting in any way [edit] as far as control flow / ergonomics is concerned?
f
yeah that kind of control flow isn't really supported w metaflow, which is a common question I get when introducing metaflow to new datascientists internally
💡 1
breaking early, conditional steps etc
having to subclass FlowSpec is a bit of a barrier for people new to python in general too since looking at a python class is more intimidating than functions (and seeing 
if __name__='main'
is going to be completely foreign + needs explanation of python internals earlier than necessary)
👍 1
so can see real benefit there in making metaflow more beginner-friendly but refactoring to make it callable seems like it'd be a heavy lift
l
Hey check this out: Flyte is apparently pushing that they have "reusable" steps in their marketing. Yeah, you can't really do that if your task function is embedded into a class 🤔. That said, I think you could argue that you can just push the core logic of that step into a library since @step-functinos are meant to be thin anyway.
I did the work to gather a bunch of DAG examples, including yours so I figured I might as well turn it into an LI post https://www.linkedin.com/posts/eric-riddoch_orchestrators-dags-pipelines-workflows-ac[…]599399329793-96VL?utm_source=share&utm_medium=member_desktop
f
ooo like that idea but with comparisons like these it’s always hard because trivial examples may not show limitations that only manifest w “production” workloads
if you’re just going by the api less so (which I think is what you’re getting at so may have jumped the gun w the last comment)
l
Haha, in the post I was kind of talking about the API, but now I'm curious about all sorts of differences. For example, the CEO of Dagster literally asked us during our evaluation of Metaflow why it was that we bought Metaflow and Dagster.
Our reasons were: • it's purpose built for DS--helps with dependency management in a way that "just build a docker image for each step" doesn't do • it allows you to specify hardware resources per step
Also the fact that ville and the others have put more thought in to DS workflows than Dagster probably has
f
yeah some of the core metaflow features like dependency management would be neat to see across other projects, vice versa too with core union/dagster etc features and how they’d be handled w metaflow
think both dagster/union have strong validation components, which w metaflow is left to the reader to implement (design choice I agree with in metaflows case)
l
"left to the reader" 🤣 So true.
f
hah that part is central to metaflow, as unopinionated as it gets
Cue the futurama neutral grey planet aliens gif
💡 1
l
Yeah, I mean. To be fair, Great Expectations and EvidentlyAI are still emerging, but are looking cool. Pretty nice that you can integrate those into Metaflow without too much hassle.
Or if they fail that we won't be fighting a decicion Metaflow made in their abstraction to favor them somehow
f
yeah bingo - lot of ways to do that too but I’ve used s3 + eventbridge to interface metaflow w things like GE and dbt. Socializing these patterns would go a long way though since on the surface it would look like metaflow would be lagging since it can’t check the box like other projects
v
re: a controller function - you should be able to hack one on top of the current class if you prefer that abstraction. You can construct a 
FlowSpec
class on the fly.
re: flow composition - we encourage sharing of business logic through modules and packages, which works beautifully today. A benefit of this approach is that it's nothing Metaflow specific: You can use modules in notebooks, test them with all standard Python tooling, package them as usual etc. There are valid cases for being able to compose a flow out of subflows. That's a feature that will land in Metaflow eventually!
f
@straight-shampoo-11124 what do you mean by 
You can construct a FlowSpec class on the fly.
?
and the hack you're describing is basically just to subclass 
FlowSpec
and implement 
__call__
?
the R package followed this functional interface pattern but not sure how you'd do it on the python side wrt constructing the graph
w
@lively-lunch-9285 / @flaky-plumber-70709 genuine question: since learning a framework implies learning few things anyway, is there really a difference between learning, say, Prefect or Metaflow style DAG (or Dagster, or Modal or Flyte)? (note that I teach Metaflow in my NYU Class to students with basic Python knowledge: if they can learn it, any professional can)
f
I think it would be easier to teach with a functional style interface. A few years ago I would have said it didn’t matter all that much but have spent a lot of time lately introducing metaflow to new data scientists and also analytics teams that only know sql; the subclassing workflow is definitely intimidating to that audience so I spend time at the beginning breaking down the basic parts of the Python file from the subclassing to what’s going on with if name == ‘main’
that being said once that’s out of the way it’s easy to explain metaflow concepts and they see the value right away (usually happens the first time they retrieve something using the metaflow client)
v
@flaky-plumber-70709 you can create a function that constructs a 
FlowSpec
class on the fly, similar to 
@pipeline
or 
@graph
w
Make sense - @straight-shampoo-11124 I'll do that as a weekend project, or assign it to my students as homework 😉
😃 4
l
Sorry, know it's been a while.
@straight-shampoo-11124 Idk that this actually matters, but isn't Metaflow limited in the types of flow logic you can do? Like, can you do statements like
Copy code
if x > 10:
    self.next(step)
else:
    self.next(other_step)
The reason I thought this was impossible was because this would make the DAG shape unknown until runtime so you probably couldn't deploy to something like argo workflows or aws step functions
In my mind, one of the features of something like a 
@pipeline
decorator is that it supports this dynamic-ness. That said, I don't know that that's a limitation that would ever really matter haha. This all feels academic.
f
yeah this isn't possible (and this kind of conditional branching is prob the #1 question I get from the data scientists that I support after they've got the basics down) I think argo workflows supports this and step functions definitely do w choice states. Imagine it may be tough to get the same functionality to be consistent w the respective metaflow decorators though
9 Views
Open in Slack
PreviousNext
Powered by