Hi all, I am trying to “dynamically” create a Meta...
# ask-metaflow
b
Hi all, I am trying to “dynamically” create a Metaflow flow in code, rather than defining all the steps by hand as methods in the Flow class. Is there a way to dynamically define Metaflow steps in code, without explicitly creating an
@step
in the source code? Or alternatively is this not a good idea? I’ll post a solution I came up with that comes close to achieving this goal, but still requires defining dummy
@step
methods in the correct order. Ideally I would like to eliminate these and rely purely on dynamic step creation. The motivation for this is I would like to be able to write reusable mixin classes to be used across multiple flows. I could then create multiple flows, each stacking together a few of these these mixins together like legos.
Copy code
from metaflow import FlowSpec, Parameter, step

class Mixin:    
    # defining this as a class attribute lets us access it without creating an instance
    method_list = [
        "method1",
        "method2",
    ]
        
    def method1(self):
        print(self.arg1 + 1)
    
    def method2(self):
        print(self.arg1 + 2)
        
class DynamicFlowBase:
    @classmethod
    def wrap_method(cls, method_str, next_method_str):
        method = getattr(cls, method_str)
        new_name = "wrapped_" + method_str
        
        @step
        def wrapped_method(self):
            method(self)
            next_step = getattr(self, next_method_str)            
            self.next(next_step)
        
        wrapped_method.__name__ = new_name
        
        setattr(
            cls, new_name, wrapped_method
        )
            
    @classmethod
    def construct_flow(cls, method_list):        
        first_step = "wrapped_" + method_list[0]
        
        # create start step
        @step
        def start(self):
            print("Running overwritten start step...")
            next_step = getattr(self, first_step)
            self.next(next_step)
        setattr(cls, "start", start)
            
        # wrap each method in method_list as a metaflow step
        for i, method_str in enumerate(method_list):
            if i < len(method_list)-1:
                next_method_str = "wrapped_" + method_list[i+1]
            else:
                next_method_str = "end"                
            cls.wrap_method(method_str, next_method_str)
       
        # create end step
        @step
        def end(self):
            print("Flow is complete!")
        setattr(cls, "end", end)
        
        return cls()

        
class TestDynamicFlow(FlowSpec, DynamicFlowBase, Mixin):
    arg1 = Parameter("arg1", default=1)
    
    # placeholder steps to make Metaflow happy
    @step
    def start(self):
        self.next(self.wrapped_method1)
    @step
    def wrapped_method1(self):
        self.next(self.wrapped_method2)
    @step
    def wrapped_method2(self):
        self.next(self.end)
    @step
    def end(self):
        pass
    

if __name__ == "__main__":
    TestDynamicFlow.construct_flow(TestDynamicFlow.method_list)
Ideally I’d like to be able to remove the dummy steps in
TestDynamicFlow
If I don’t include the steps, I get the following error
Copy code
Based on static analysis of the code, step start was expected to transition to step(s) end. However, when the code was executed, self.next() was called with wrapped_method1. Make sure there is only one unconditional self.next() call in the end of your step.
v
great question! When it comes to composing flows and the business logic they contain, there are three major approaches: 1. 🟢 Plug-in business logic, i.e. modifying on the fly what happens inside steps 2. 🟡 Dynamically composable decoupled flows 3. 🔴 Constructing a flow DAG dynamically The first approach works well today. It is a widely used pattern. See a simple example here (a pluggable algorithm in a module) and a more complex example here (dynamically defined models and feature encoders). The second approach is used widely at Netflix and will be available in OSS soon as we'll release support for event triggering. The third approach, which I understand was your original question, is not supported today. We have plans to support flow composition so that you can construct a flow from multiple sub-flows, but even that won't allow altering the flow structure on the fly. If you need that level of dynamism, then the second approach is your best bet.
s
while you can't change the DAG itself dynamically, you can certainly alter its behavior on the fly. Take a look at this
@skip
decorator as an example of how to do it https://outerbounds-community.slack.com/archives/C02116BBNTU/p1660145168960519?thread_ts=1660134373.601939&amp;cid=C02116BBNTU
b
Thanks for the resources! I’ll take a look at the second approach and see if I can get it to work for my use case. The @skip decorator is great, and coincidentally I am already using it in my flows!
v
great! Take a look at the event-triggering memo and see if it works for you. It'll be officially out maybe in a month or so but we are happy to help you explore it even before if you are interested