is there a way to auto batch workers? eg i'm running a flow in a resource constrained environment, that can only run up to 150 workers at a time. using

foreach

though would result in 1,500 workers being run concurrently, so just setting

--max_num_workers

to 1,500 runs into OOM errors. has it been considered to allow an option to set batch sizes for the

foreach

workers? eg run

X

number of workers at a time, until the task is done, then it moves to the next step?

I have hit a similar issue by myself. Currently there's no way to do auto-batching like you described. A workaround is to use an environment, e.g. Step Functions, that can handle a higher

--max-workers

(or just wait longer with a lower

--max-workers

). Are these local tasks or

@batch

?