Hello, I had a question about how

parallel_map

behaves to see if my understanding of what I am seeing is correct, and ask if there are any workarounds. It appears to me that when passing a large iterable (much larger than the number of available cores) to

parallel_map

the processes are run in batches rather than streamed. For clarity here are my definitions of batching and streaming: • batching: run

n

processes, wait for all to complete completion, and then run the next

n

processes • streaming: run

n

processes and run the next

1

process as resources become available I think

parallel_map

is using batching because I see all

n

processes begin, then see a slow reduction in the number of active processes, and then eventually a flare back up to

n

processes to repeat the loop. I am working with extracting spatial data over property boundaries with very variable sizes. With the batching behavior a single large property can effectively block computations on all

n

cores if it takes significantly longer than all the other properties in its batch. Streaming (like metaflow does for individual workers across resources) would mean less work trying to balance batches. I currently shard a large set of properties into smaller sets that can each be sent to a worker on AWS Batch. Within these sets of properties it is most efficient to process on individual properties and polygons, but because I am using Batch I do not want each property to be a task that ends up in the batch job queue. So far I have tried or considered: • Splitting properties into equal sized units (workable but this is introducing complications elsewhere in my flow) • Moving to

Pool().map

, but I don’t know if that has the same batching behavior (I am more familiar with multiprocessing in R, where streaming is the norm in the

furrr

package) • Doing something wild like splitting this step into its own flow and calling it via subprocess within each Batch worker to use the available cores for individual tasks within a worker (effectively set max-workers in the parent flow to control number of Batch workers, and then max-workers in the child flow could use all resources available) Would appreciate any tips or other approaches to parallelize this step that will be a bit more efficient at using the available cores.

I believe in distributed computing... "map" usually means batch. If you want streaming like behavior, modern python comes with "Executor" (https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) that allow you to

submit

tasks into a queue. The underlying pool of workers will pull work from the queue. This is more efficient in terms of core util vs map (for varying, independent task sizes). However if you need a MapReduce pattern, then map is a simpler pattern (since it maps directly to what you need )

you can try the approach that Jackie suggested, or here's another idea that you can try:

parallel_map

actually does your streaming semantics but a bit inefficiently. If you can estimate the amount of the work in advance, sort the list passed to

parallel_map

in the ascending order in terms of the workload size. It should keep all cores busy constantly, instead of the "flare-up" behavior you described

I think executor is exactly what I need for this step. I worked a little on packing the properties into a more uniform order, but the differences can be orders of magnitude between individual properties even when sorted. Splitting up the larger properties gets me a little closer, but eventually the proliferation of smaller polygons creates bottlenecks upstream of this process

Alrighty. FYI executor has “map” function too, which can offer a similar higher level UX. Parallel map offers some subtle pluses though, like lambdas can be passed as the fn, executor cannot do that. May not matter for you.

The executor map function seems to be just what I need and is almost a near drop-in for what I had in place. I did have to create a wrapper function around my call to avoid passing in a lambda, but that’s light work. Just need to handle memory now. My tests so far have seen some memory bloat, which I think is related to long-lived executors. We’re not on 3.11, but it looks like

max_tasks_per_child

add in 3.11 might solve that problem for long lived executors