Hi Folks, I'm a cheminformatician who's a novice at Metaflow, but I'm loving it so far. I just had a quick question: Is there any way I can use parallel_map or something similar to distribute compute locally? I have a simple function (RDKit's MolFromSmiles, which reads an ASCII string and turns it into a molecule, in case anyone is wondering) that I need to get running across a list of a few million strings. I have a beefy workstation that makes quick work of it when I use a Pool (outside of metaflow). parallel_map would've been the ideal, but I see it only works with batch or kubernetes. Joblib and multiprocessing.Pool obviously don't work due to them not playing nice with pickle. Has anyone faced a similar issue? If so, how did you overcome it? I'd be grateful for any help. Thanks in advance.

You can use parallel_map outside of metaflow as well

also note that if you do a normal Metaflow

foreach

and run it locally on your beefy workstation, it'll use multiple CPU cores on your system (you can control how many with

--max-workers

)

Thanks for the quick reply. I meant parallel_map seems to only use one CPU core when I'm using it on prem on my workstation with 64 cores. So I guess I'll have to split it into foreach then, right?

it should use multiple cores. Did you give it a list with more than one item to process?

Yeah. It's a Pandas Series with 1.5 Million items. Should I convert it to a list first?

yep! Also if it doesn't take a long time to process one element, it's more efficient to chunk it to short lists

Alright. Got it. Thank you so much for the help :)

cool! Feel free to follow up here if it doesn't work as expected or if you have other questions