Hello all, I am pleased to announce that @User and I collaborated in developing a

@ray_parallel

decorator, and it's now available to be used as a Metaflow extension. The

@ray_parallel

decorator spins up an ephemeral Ray Cluster that can be used to run Ray applications. This was a labor of love to get Metaflow integrated with Ray. Thus far, we've tested it on a variety of different applications, such as: Fine-tuning LLMs with Ray Train, Hyperparameter Tuning with Ray Tune, Distributed XGBoost, etc. Due to the transient nature of the cluster, only Ray Serve would not be compatible. Major thanks to @User and @User for significantly enhancing the UX of the original version of the decorator! We would welcome and very much appreciate any feedback on this decorator - please give it a try yourself! See here: https://github.com/outerbounds/metaflow-ray