The human-centric platform for production ML & AI

Outerbounds

Does Metaflow support orchestrating ML solely within Databricks? (eg: using Databricks workflows, or Airflow to orchestrate).  I found a lot of questions in this channel related to integrating with Databricks with say @pyspark decorator, but am curious about running Metaflow within Databricks.

I ask as Databricks is becoming the the paid platform for many company data solutions (eg: ETL with Spark and DataLakes).

thank you!

out of the box `@pyspark` works with Servlerless EMR but I think some folks have used it with Databricks too.

Another great way to integrate the two is via Iceberg, along the lines what Netflix does (see <https://netflixtechblog.com/supporting-diverse-ml-systems-at-netflix-2d2e6b6d205d|Fast Data here>)

Iceberg works with Databricks? Is this using the Unity Catalog Open APIs?

<https://docs.databricks.com/en/delta/uniform.html|it's a quickly evolving story>

<https://www.databricks.com/blog/databricks-tabular|they acquired> our ex-colleagues who created Iceberg at Netflix last summer