Looking at the example where 2 models are trained in parallel: https://outerbounds.com/docs/intro-tutorial-S2E3/ Could one also do this “dynamically” in parallel. Let us say we split a data frame into N folds and we want to fit a model for each fold where N could be passed to the flow as parameter?

Yes, say you have N chunks of your data stored in

self.data_chunks

, you can do:

self.next(self.train, foreach='data_chunks')

and you will have N parallel

train

steps spawn with each of them receiving one chunk of the data. You don’t even need to declare N while starting the flow. Meaning, even if N was something that was calculated at runtime, the dynamic split would work. Let me know if that helps answer your question or if there’s anything else I can help with!

yes thanks. I tried something similar but as I still run locally it crashed my machine. curious how to you access the specific chunk in self.train? Via:

self.input

?

So you’ll have

N

unique instances of the

train

function. Each instance will receive one chunk of the data in its

self.input

. Therefore for each instance of train

self.input

is just one chunk and you don’t have to access specific indices yourself.

yes thanks. that’s what I have done/used.