https://outerbounds.com/ logo
Join Slack
Powered by
another question I have is there a common pattern ...
# ask-metaflow
m
another question I have is there a common pattern for combining flows together?
🙌 1
say that I have two flows A/B, and I want to define another flow C that can act like some kind of "flow runner", and I'd like to be able to pick flow A or flow B
r
I have the same question - I'm trying to figure out how best to combine multiple flows in different workflow engines, and wonder what everyone is doing... (e.g. if they combine them in Airflow, or deploy flows as ECS tasks, and then write their own orchestration logic for invoking those tasks with say, EventBridge)
m
yeah I guess you could orechstrate flows but it'd be nice to have 
metaflow
handle all of that itself
one of the tutorials kind of has an example of that, where you load the outputs from another flow and manipulate them inside that flow, but that doesn't actually run that other flow
f
the native way to do that within metaflow (if you're running on k8s) is to use the 
@trigger_
class level decorators: https://docs.metaflow.org/production/event-triggering/flow-events
👀 2
m
but that's if the flows are deployed, right?
what if I'm just developing stuff locally and want to just test my "flow of flows" locally?
oh wait; there's a section right at the bottom here
f
yeah but wouldn't treat them being deployed as the same meaning as "production" or anything - so long as you have metaflow running on k8s there's no real barrier to creating a workflow from that and treating it like it would be local (but you have to have the dependencies worked out with @conda or the new pip dec)
I generally recommend getting flows deployed sooner rather than later because it opens up a lot of functionality/patterns that you dont have locally
🙌 1
m
one more question; is there a way to define things in a "pull forward" kind of way?
say that A triggers B and C
and I want to run C
f
for @refined-pilot-33461's q around interfacing with other systems, usually I throw s3 or something in as the intermediary and then trigger metaflow on s3Put notifications or eventbridge etc
👀 1
so by running C you want to ensure that A & B both run?
m
no I just want A to run
f
ah - yeah IIRC don't think theres native functionality there but makes sense, to do it would basically deploy a new version of A that only has C as a trigger (with --branch maybe identifying that its just going to trigger C)
I'm still in the stone age though running metaflow on batch and have to define my triggers using eventbridge/s3
what you want is basically dbt's graph operator functionality: https://docs.getdbt.com/reference/node-selection/graph-operators
👀 1
just in metaflow that is
c
is there a way to define things in a "pull forward" kind of way?
so by running C you want to ensure that A & B both run?
you could have flow 
A
publish event 
e
, assuming results in 
A
meet some success criteria, and then 
@trigger
flows 
B
and 
C
based on event 
e
.
note the above is an argo-specific thing - luddites like bryan will need more DIY approach 😉
f
then you'd manage multiple publish events though right, for every downstream flow?
and mix conditional logic in which could get ugly quick with ragged/complex dependencies
c
you could do it that way, or publish the same event and then have multiple flows/consumers react to that single event from 
A
5 Views
Open in Slack
PreviousNext
Powered by