Hi, does anyone know if Metaflow is compatible wit...
# ask-metaflowf
few-autumn-83204
12/14/2022, 6:51 PMHi, does anyone know if Metaflow is compatible with Databricks?
✅ 1
v
victorious-lawyer-58417
12/14/2022, 6:53 PMhi Fabricio! Metaflow works nicely with Spark, especially when it's used for ETL and large-data preprocessing for ML workflows, so using Databricks and Metaflow side-by-side is a sensible idea
f
few-autumn-83204
12/14/2022, 6:56 PMHi Ville, thanks for your answer, i am reading your book (currently chapter 4) and i am finding it very useful. Is there any tutorial in order to leverage Metaflow’s possibilities in DataBricks?
v
victorious-lawyer-58417
12/14/2022, 6:58 PMread the chapter 7 that's about data - most of those ideas are applicable to Spark with Databricks
victorious-lawyer-58417
12/14/2022, 6:59 PMwe have an experimental `@pyspark` decorator that's tuned for Spark-on-EMR but the same idea could work with Databricks too. You can see an example here https://github.com/outerbounds/metaflow-pyspark/blob/main/example/sparkflow.py#L10
f
few-autumn-83204
12/14/2022, 6:59 PMThank you very much!
v
victorious-lawyer-58417
12/14/2022, 7:00 PMlet us know how it goes! We are happy to help you get going with it.
We have plenty of new documentation planned for data topics next year but meanwhile we can help you here in person 🙂
🙌 1
f
few-autumn-83204
12/14/2022, 7:02 PMGreat, I’ll let you know as soon as I have any feedback. Good to know and thanks for your help!
👍 1
32 Views