Hi, All. I have a question about using Runner/Trig...
# ask-metaflow
h
Hi, All. I have a question about using Runner/Trigger to run a series of flows. Here is the problem. I have a parent experiment that needs to 1. run a series of child experiments. each on a set of days. 2. summarize all these experiments, write some report these child experiments can be completely paralleled. • I may use Runner, but the doc says it is equivalent to local runs, which doesn't work for me in production. • I may use trigger. but then it's hard to test locally. any suggestions on what should I do?
1
one solution: I will not construct child experiment as flow, but just steps in the parent experiment
d
I am not 100% sure I understand what you are trying to do. To rephrase, are you trying to have a parent experiment A run daily and on each run of A: • it runs a set of sub metaflow flows B1, B2, B3, … • waits for the results of those • inspects the results and writes a report?
h
thanks for the response. the parent experiment is usually an offline study. say it wants to study data from 2024/01/01 - 2024/12/31. each child is one month study (e.g. 2024/01/01 - 2024/01/31). the parent basically 1. create some parameters for each child on the fly. 2. let them run. wait for their finish. 3. collect all their results, summarize
sometimes we will kick off multiple parent experiment. each one has different parameter values, but focus on the same period (e.g. one year)
d
You could use the runner for this for sure. You can launch non local runs with it (using batch, kube etc). That should allow you to run it locally (for testing) and as part of a more production type of run. Of note, the node running the parent experiment would have to monitor a bunch of runs so that may get a bit heavy. You could spread it out in à foreach and have those monitor your runs too.
👍 1