Does

@torch_run

support kubernetes? In the post from last year it says not yet. https://outerbounds.com/blog/distributed-training-with-metaflow/ But I think I see support in the repo here https://github.com/outerbounds/metaflow-torchrun

yes, it does now. you would need to install jobsets on your kubernetes cluster and if you are running at scale, kueue as well.

What do you mean "scale" exactly,

to get access to true gang scheduling semantics, you would need both jobsets and kueue. just using jobsets can result in deadlocks - the probability is rare if you are not running many workloads and have ample compute capacity available.

Interesting I definitely haven't done that yet so.