I guess Ray integration is only for AWS Batch not for K8? Can we use Ray with Metaflow if we setup Ray's cluster on K8?

Correct. There is an experimental version of Metaflow that enables the metaflow_ray decorator and other gang scheduling decorators on kubernetes, but stable version of Metaflow is batch only. Alternatively, if you already have a Ray cluster you could submit a Ray job to the cluster from a Metaflow task. Can chat sometime next week on either pattern if you’d like.

Would be great, how is Tuesday morning EST time?

10am EST?

Invite is for 11EST, it's fine as well if it works for you

Apologies, resent!

Morning, we are on the call now.

Any update on if / when this will come into the stable version of metaflow?

Hi Christian, we are in the process of merging in this PR for @parallel (and downstream decorators like decorators like @metaflow_ray, @deepspeed, @torchrun) with @kubernetes.