Hi, I want to run a multi-node training job using Metaflow and I intend to use the parallel decorator and I have setup my flow similar to the flow and training code found here: https://github.com/rileyhun/llm_finetuning_metaflow/tree/main/pytorch-deepspeed. Currently we are using Kubernetes cluster and I'm using the Kubernetes decorator to provision my compute. However, when I use the the parallel decorator with the Kubernetes decorator, I get the following error:

Kubernetes error:
    @kubernetes does not support parallel execution currently.

I was able to trace the error to https://github.com/Netflix/metaflow/blob/0acf15a0efaa51a5421e953ae8e587f734539618/metaflow/plugins/kubernetes/kubernetes_decorator.py#L219 and this indicates we can't use the parallel decorator or any of its variants (pytorch_parallel or ray_parallel) with the Kubernetes decorator. I'm wondering what is required to get support for multi-node training with the Kubernetes decorator? I see that for the AWS Batch decorator, there is a function specifically for setting up the here: https://github.com/Netflix/metaflow/blob/0acf15a0efaa51a5421e953ae8e587f734539618/metaflow/plugins/aws/batch/batch_decorator.py#L360. Would we need to do something similar for the Kubernetes decorator to get the parallel decorator supported? Thanks!

we are working on experimental support for this now, but it is not in the most robust state. please DM if you'd like to try it out and/or develop with us!

Hey Eddie, we have been primarily using Pytorch for our model development. Specifically, we are using Deepspeed and Pytorch Lightning/HuggingFace trainers to train models.

Right on, looking at deepspeed actively. Want to have a chat on Friday about where things are currently re: Kubernetes + parallel and roadmap?

Sure @flat-television-23413. I'm also free earlier this week as well!