https://outerbounds.com/ logo
Join Slack
Powered by
Hey folks, Does anyone have any recommendations o...
# ask-metaflow
a
Hey folks, Does anyone have any recommendations on what is the maximum amount children one can have when using 
foreach
on Metaflow? A user wants to process ~2M files, and they're already doing some batching and then each batch will take advantage of 
parallel_map
. So wondering what would be a good balance between the horizontal fan out and the batch size?
✅ 1
v
if you want to maximize throughput, it probably makes sense to do at least a few minutes of work per task (or maybe more). Each task has some amount of overhead to launch, so if the amount of work is lower per task, then the overhead-workload ratio might get too low
so if processing one file took a second, 5 minutes per task would mean a batch size of 5 * 60 = 300 files / task, so 6666 tasks in total, which is a reasonable order of magnitude
your 
--max-workers
is likely to be much lower than 6000, so there isn't much downside of doing larger tasks probably
a
Awesome, I think we were heading in that direction too! Just wanted to make sure there aren't any inherent gotchas before we proceed. Thanks Ville!
👍 1
v
it’d be great to hear how it goes. Other people here might be curious too
❤️ 1
a
CC - @sparse-florist-36640 ^^^
Hey @victorious-lawyer-58417, I think Thomas is running into OOM errors by packing too many files in each batch that gets passed to the 
foreach
children? I think Thomas can circumvent this issue by decrease the batch size, and he's switched to high memory instance types, but we probably still want to have a reasonable batch size to process the 10M files faster. Any suggestions or recommendations on things to consider?
We are going to increase the EBS volume and increase the memory in the 
@batch
decorator. But besides this, Would any of the following params in the 
@batch
decorator: 
sharedMemorySize
, 
maxSwap
and 
swappiness
help to alleviate OOMs as well?
3 Views
Open in Slack
PreviousNext
Powered by