Has anyone developed a Metaflow Ray plugin? I’m looking at supporting Metaflow on Databricks. Their options for orchestration is: • Ray • Airflow • Spark • Very sparse documentation for Databricks Workflows

The Airflow and Spark integrations exist today. I am curious how would you like to use Ray within Databricks.

The Spark integration isn’t a Metaflow orchestrator… and I’m afraid Airflow doesn’t truly support foreach with a multi-step split. Ray could be the orchestrator https://docs.ray.io/en/latest/workflows/basics.html

I am not aware whether databricks provides support for hosted Ray

Also, @hallowed-glass-14538 has been working with the Airflow team to bring nested foreach support to Airflow

• Databricks just added: https://docs.databricks.com/machine-learning/ray-integration.html • And https://www.anyscale.com/ is founded by the executive chairman of Databricks