https://outerbounds.com/ logo
Join Slack
Powered by
I'm looking to run a flow within a flow and pass c...
# ask-metaflow
p
I'm looking to run a flow within a flow and pass custom parameters into the child flow from the parent flow. I was looking for something like 
NBRunner
but in a 
my_parent_flow.py
setting
Copy code
params = {"alpha": 1, "beta": True}
run = NBRunner(MyFlow).nbrun(**params)
I tried to do it in the init method of my child flow like this
Copy code
class ChildFlow1(FlowSpec):
    def __init__(self, parameters):
        self.alpha = Parameter(
            "alpha",
            help="Alpha parameter for ChildFlow1",
            default=parameters.get("alpha", 1.0),
        )
        self.beta = Parameter(
            "beta",
            help="Beta parameter for ChildFlow1",
            default=parameters.get("beta", "default"),
        )

    @step
    def start(self):
        self.result = f"Result from ChildFlow1: alpha={self.alpha}, beta={self.beta}"
        self.next(self.end)

    @step
    def end(self):
        print(
            f"ChildFlow1 completed with parameters: alpha={self.alpha}, beta={self.beta}"
        )
Then I run the child flow from my parent flow like this:
Copy code
child1_run = ChildFlow1(parameters={"alpha": 2.5, "beta": "custom"})
child1_run.run()
self.child1_result = child1_run.result
However, I'm seeing a cryptic 
RecursionError: maximum recursion depth exceeded
error. Any pointers in the right direection?
✅ 1
a
can you try
Copy code
class ChildFlow1(FlowSpec):
        alpha = Parameter(
            "alpha",
            help="Alpha parameter for ChildFlow1",
            default=1.0,
        )
        beta = Parameter(
            "beta",
            help="Beta parameter for ChildFlow1",
            default="default",
        )

    @step
    def start(self):
        self.result = f"Result from ChildFlow1: alpha={self.alpha}, beta={self.beta}"
        self.next(self.end)

    @step
    def end(self):
        print(
            f"ChildFlow1 completed with parameters: alpha={self.alpha}, beta={self.beta}"
        )
d
small correction, it’s not 
self.alpha = Parameter
but jsut 
alpha = Parameter
(same for 
beta
)
👍🏼 1
(parameters are just class variables)
a
fixed!
p
thanks! trying this, but how can i pass parameters from parent to child flows this way?
running into a new issue. here's my parent flow in `hierarchy.py`:
Copy code
class ParentFlow(FlowSpec):
    @step
    def start(self):
        self.next(self.run_child_flows)

    @step
    def run_child_flows(self):
        namespace("default")

        # Execute ChildFlow1 with custom parameters
        child1_run = ChildFlow1()
        child1_run.run()
        self.child1_result = child1_run.result

        self.next(self.end)

    @step
    def end(self):
        print(
            f"Results:\nChildFlow1: {self.child1_result}"
        )
Copy code
2024-09-05 17:01:23.833 Workflow starting (run-id 32):
2024-09-05 17:01:24.521 [32/start/1465 (pid 1297796)] Task is starting.
2024-09-05 17:01:25.795 [32/start/1465 (pid 1297796)] Task finished successfully.
2024-09-05 17:01:25.957 [32/run_child_flows/1466 (pid 1297819)] Task is starting.
2024-09-05 17:01:26.661 [32/run_child_flows/1466 (pid 1297819)] Invalid command:
2024-09-05 17:01:26.661 [32/run_child_flows/1466 (pid 1297819)] Step run_child_flows doesn't exist.
2024-09-05 17:01:27.037 [32/run_child_flows/1466 (pid 1297819)]
2024-09-05 17:01:27.079 [32/run_child_flows/1466 (pid 1297819)] Task failed.
2024-09-05 17:01:27.206 Workflow failed.
2024-09-05 17:01:27.206 Terminating 0 active tasks...
2024-09-05 17:01:27.206 Flushing logs...
a
to execute
Copy code
# Execute ChildFlow1 with custom parameters
        child1_run = ChildFlow1()
        child1_run.run()
you would need to use the NBRunner or the Runner API
p
I tried the NBRunner from within my script but here's what I see:
Copy code
child1_run = NBRunner(ChildFlow1)
child1_run.nbrun()
Copy code
metaflow.runner.nbrun.NBRunnerInitializationError: 'NBRunner' requires an interactive Python environment (such as Jupyter)
I will try the runner API and report back
b
@plain-barista-93517 NBRunner is to be used inside notebooks
👍 1
use the Runner API for scripts
@plain-barista-93517 let me attach an example
place them in the same directory and do 
python parent_flow.py run
child_flow.pyparent_flow.py
🙌 1
to pass your own parameters, the only thing you need to do is: 
child_runner = Runner("./child_flow.py").run(alpha=2, beta="pavan")
p
super helpful, and this works great, thank you! quick follow up: if i do 
python parent_flow.py run --with kubernetes:image=custom_image
will the child flows be in the same environment as the parent?
b
hmm, the child flow will need to have 
--with kubernetes
as well this can be done with 
decospecs
I guess, see the example which has
Copy code
Runner("myflow.py", decospecs=["kubernetes:gpu=4"])
https://outerbounds.com/blog/metaflow-in-notebooks-and-scripts/#the-new-runner-api
thankyou 1
otherwise the child flow is just another subprocess within the same pod / job of the parent
✅ 1
p
Nice, will try this. It seems like the default behavior (i.e. without 
decospecs
) results in a successful run of the parent flow within the same kuberenetes pod
b
yup, exactly
behind the scenes, Runner = CLI only, but in a subprocess
🙌 1
but with all the goodies like type checking on the fly, user defined parameter injection, etc.
👍 1
p
thanks for your timely help!
👍 1
9 Views
Open in Slack
PreviousNext
Powered by