Hello, Long time user of Metaflow here. We've got Metaflow set up with AWS Batch and it's running smoothly with the Datastore on AWS S3. I can run jobs from a single on-prem node and scale up to AWS Batch as needed. I'm aware that Metaflow can be integrated with Kubernetes for on-prem nodes, as mentioned in Right now, we're using on-prem nodes managed by a SLURM scheduler. Since we need to utilize these on-prem nodes and managing them with Kubernetes isn't an option, I'm wondering what it would take to get Metaflow working with SLURM. Is this something I could realistically do on my own? I'd appreciate any guidance you can give. If it's going to be a big project, please let me know so we can look into other options.

Hi! A quick way to do this if you need to run on slurm is submit the job with

srun

from a subprocess in Metaflow task. That forgoes many benefits of using Metaflow, however, so you'd need to manage slurm nodes more manually. A deep integration like @slurm would be a bigger lift. some incomplete thoughts about questions we'd need to build and document solutions for: • how to run the tasks in containers on slurm nodes? ◦ how to distribute dependencies in @conda and @pypi packages Metaflow builds to slurm nodes? • how to assign slurm cluster priority to jobs submitted from metaflow tasks? • are there issues with auth from metaflow metadata store and nodes? • what is the story for moving/mounting large datasets on the slurm nodes? • how to unify Metaflow's remote task lifecycle management with slurm jobs? ◦ @retry, --resume, @catch, etc. and the RUNNABLE, STARTING, RUNNING states • how to set up internode comms? does this happen once at slurm cluster setup like is typical, or does it happen dynamically like how @kubernetes configures multinode comms when used with @parallel? I'd estimate several months at least to get @slurm sufficiently reliable for large scale jobs.