The human-centric platform for production ML & AI

Outerbounds

I've got a question about best practice.  I have a flow where the a significant portion of steps involve running command line tools.  Prior to Metaflow, this would have been designed as a script or series of scripts that would have been distributed to multiple nodes using a workload manager like slurm.  When converting this to Metaflow, I assume there aren't really any alternatives to just calling `subprocess` from within the step?

hi Tim! If the scripts are in Python, you don't necessarily have to call them as subprocesses

Also if the scripts call C libraries, it may be possible to call them directly from Metaflow tasks

calling a subprocess has some benefits though: In particular, you can handle failures in the subprocess gracefully in the Metaflow task, so you get all logs &amp; artifacts even if the process crashes hard

in similar situations in the past, I've seen folks starting by migrating existing scripts as subprocesses initially, making sure that you get the whole workflow running end-to-end with Metaflow.

You still many benefits of Metaflow when it comes to tracking &amp; versioning, data loading, and compute.

Over time, you can migrate scripts to Python tasks, if it makes sense.