Hi 👋 I have a question regarding the spinning up of pods when using

foreach

. I have a step that is using the

self.next(self.next_function, foreach='some_variable')

. When looking at the timeline of the

next_function

I see that not all childs are spinning up at the same time (take a look at the picture attached for the gradual spin up) (I have set the

max-num-splits

to the num of the produced childs) Is there any configuration that I’m missing so all pods will spin up in parallel and not in batches?

Are you running this on AWS Batch or K8s?

We’re running in K8s (AKS) in Azure

@dazzling-van-84908 for any time slice you should see 200 tasks running. Can you confirm that is the case?

Thanks @ancient-application-36103 We do see 200 tasks running, but as @bitter-orange-8782 mentioned above with adding more max-workers we see larger time gaps between batches which effectively negate the effect of the addition. We want to understand why is this happening.

As long as you have sufficient compute capacity, there shouldn’t be any additional delay.

Compute in the workers themselves or in the application server?

the workers

thanks @square-wire-39606, but something is still not clear to me, when we are starting a new batch metaflow is initiating each worker in a serial manner, so in my pic when worker number 907 has finished his processing i would guess that it will start worker number 908 immediately and if it takes time for that worker to spin up then "it will just take time" and we will see it in the UI as long execution of the task - not delay between batches. (also, worker number 908 should start immediately as all the previous workers has finished processing and their state is `complete`in k8s, with scale 0 - so all resources are free again).

are there no other tasks that executed in between task 907 and task 908?

No, there aren’t @ancient-application-36103 Some of the tasks show that they are still executing in UI but it seems like a bug. In reality we know that all tasks are complete after ~25 secs, This is what we see in k8s, logs and the UI itself after the task is done. For example: In the UI we see that task X shows as executing for a couple of minutes, but when it is done it shows that execution time was 25 secs. This happens only if we increase --max-workers to 200. There is no such lag with default num of workers.

if you increase parallelism, the UI has to do more work to track multiple tasks to paint this view - if you try moving to a large instance that hosts the UI service, then this issue should dissipate.