Awesome to see official support for distributed training! We are using Metaflow on AWS but we are planning to have a hybrid approach to use local K8s cluster with GPU for alternate compute. The idea is to have common metadata service, UI, storage on AWS. So I have questions. • One of the main usecases for local k8s is distributed training, is there any ETA for the new features to support K8s as well? • I see the hybrid approach to be quite possible, but not really sure. Are there any challenges here? Thanks a lot 🙏

No official ETA for supporting @kubernetes with @parallel yet. We are working on some POC implementations I could show if you are feeling adventurous or want to define requirements and build it with us. Outerbounds is also exploring the hybrid approach you suggest. Feel free to DM if you'd like to set up a demo of the POC / chat about either piece 🙂

Thanks @flat-television-23413 for the quick response. Sounds great, I do not have clear idea about k8s or doing multi node with it. However I am interested to understand the hybrid approach, as its more of a necessity for my org in the near future. I will reach out in your DMs once I get a better understanding of our requirements. I thought it was already possible and that many teams are doing so. Am I wrong?

Not sure how paved the path is for Metaflow deployment in AWS with local Kubernetes compute at the moment, but it can definitely be done.

Great, I should be able to try that soon. If and when I do I shall share about it here. Thanks @flat-television-23413!

Nice, keep us posted!