So I intended to run this “pseudo code”, which is hopefully self explanatory, in parallel using --batch

…
self.next(self.train_and_score_model, foreach='fold_number_list')
…
@step
    def train_and_score_model(self):
…
@step
    def join(self, inputs):
        print("Rejoining from the folds.")
        self.next(self.end)

I only realised that AWS batch works sequentially using a queue as far as I know. Curious, how can I run train_and_score_model in parallel for all folds in fold_number_list? Can I not utilise the several vCPUs as well in AWS batch as stated in the resources not available locally? python Bla.py --with batch:cpu=16,memory=30720,queue=amazing_queue --environment=conda run

yep, that’s the right pattern! As long as you have enough capacity in your cluster, it’ll keep scheduling tasks from the queue for parallel execution

Thanks - but is the queue not defeating the intention of parallelism in this case. Ideally, I want to run all N folds “truly” in parallel. I will have a look at what you shared. Thanks!

tasks won’t stay in the queue. Just increase the —max-workers setting to be above the number of tasks you want to execute in parallel

ah so stuff is taken from the (aws batch) queue as resources become available? so if I use:

--max-workers 11

and I have 10 fold that ensure most likely parallelism?

precisely!

thanks. reading more thoroughly max-workers is defaulted to 16 anyway …

try

python bla.py --with batch:cpu=2,memory=3840,queue=xxx --environment=conda run --max-workers=11 --max-num-splits=11

thanks so after run .. tried a few versions but this does work!