The human-centric platform for production ML & AI

Outerbounds

I am using Metaflow to run a process locally on a computer with 32 GBs of memory, 32 GBs of swap, 24 cores, and 1 GPU with 12 GB of memory. I individually specified varying resources, such as @resources(cpu=1, memory=8192), above each @step. However, Metaflow still starts 16 processes for some reason, using all of the cores, and causing the system to run out of memory. Based on the setting I have, the most processes it should be able to start is 8ish; so, I'm a little confused how it is managing to start 16. Any suggestions?

`<https://docs.metaflow.org/api/step-decorators/resources>` I think resources only work with batch or k8s, not yuor local machine

if you want to limit parallelization on a foreach, you can do: --max-workers k, and all parallel steps would be limited to at most k at a time

Interesting, I was reading a few thread on here and thought it would work for local too. Thanks for the info. `--max-workers` looks like it should work! Thanks!