Hi, looking for a solution for skipping steps base...
# ask-metaflowh
handsome-ocean-90256
09/28/2023, 8:14 AMHi, looking for a solution for skipping steps based on certain condition, for example if condition is True do step a otherwise step b, is that something that metaflow can do?
✅ 1
d
dry-beach-38304
09/28/2023, 8:30 AMnot the way you are mentioning it. You can do that within your step but not at the metaflow DAG level (the graph is “static” except for the dynamic number for foreach nodes).
h
handsome-ocean-90256
09/28/2023, 8:50 AMMy use case is that I have one step for processing a small file and another for processing a large file, each step with different resources, if we launch the large step even if we don't do anything within in wouldn't that still use the large instance unnecessarily?
handsome-ocean-90256
09/28/2023, 9:02 AMI just saw an older thread, with using a skip decorator, I think that does the trick!
d
dry-beach-38304
09/28/2023, 9:04 AMthe skip decorator wraps what I mentioned basically.
dry-beach-38304
09/28/2023, 9:08 AMto be very clear: something will execute (so if you are using batch for example), there will be a batch node that is allocated to evaluate the condition and “skip” the main computation. It is equivalent to doing a
Copy code
if not skip_condition:
# Do step
self.next(...)h
handsome-ocean-90256
09/28/2023, 9:51 AMMakes sense, thanks!
f
flat-hospital-6180
09/28/2023, 6:24 PMI think what maybe more elegant is something akin to the "BranchOperator" of Airflow @handsome-ocean-90256 - https://www.analyticsvidhya.com/blog/2023/01/data-engineering-101-branchpythonoperator-in-apache-airflow/ this can be a good-to-have feature for Metaflow, @dry-beach-38304
h
handsome-ocean-90256
09/29/2023, 8:01 AM@flat-hospital-6180 yeah I used Airflow in the past, this was very useful indeed, would be nice if metaflow can have something similar. :)
f
flat-hospital-6180
09/29/2023, 8:54 AM@victorious-lawyer-58417 @ancient-application-36103 let me know if you nod at it, and I can give it a try ...
my idea, taking the ("inspiration" sought, not just idea copy-paste 🥹) from Airflow Branch Operator, shall be rather a "switch operator"(decorator basically) , vis-a-vis
Copy code
@switch(
cases: [
("<var1>", "<comparatorN>", "condition1"),
...
("<varN>", "<comparatorN>", "conditionN"),
],
actions: [
(<task1> or <action1>,
...
(<taskN> or <actionN>,
]
default: <taskDefault> or <actionDefault>,
switch_engine: <switch- action>
)
cases: list of 'cases' wherein case is a "tuple" / Pydantic Object/schema of what-to-compare, to-how-compare, compare-with
actions: list of 'actions' wherein action is a task a.k.a self.next(...) or an "action" vis-a-vis a method that is a custom delegation of decision-making in case of the condition
*length of 'cases' and 'actions' array need to be same. Ofcourse, if no condition met, a 'default' task or custom method, must kick in.
Alternatively, one may just specify a "switch_engine" implementation, that decides for itself and yields a final task to be executed next.
* both switch_engine and (cases, actions, default) cant be Non-empty at the same time, still in case, the "switch_engine" gets the precedence.s
square-wire-39606
08/27/2025, 9:30 PMconditionals are now generally available
11 Views