Hi team! I'm diving into Metaflow and have a question for the experienced users here. I've read almost all the documentation and tried some examples but still trying to understand the core problem it was originally designed to solve and its underlying philosophy. From a practical standpoint, what are the compelling reasons for mle\ds\mlops to choose Metaflow if you're currently Airflow user as an example? I've mapped out our typical ML development lifecycle (from experimentation to production), and I'm finding it hard to pinpoint specific stages where Metaflow offers a significant improvement in convenience or capability but i believe it should exist.

good question! Much of it boils down to ML/AI/data science vs. data engineering being similarish but separate disciplines: Data eng • Workflows are DAGs of data processing steps, often expressed in SQL - relatively homogenous structurally • Compute happens in the data lake / warehouse • Development is much about ensuring that SQL produces the correct results which you can do in a notebook etc • Software environment is homogeneous: mainly connect to the data warehouse • Software dependencies are more limited: Maybe you need DBT, Great expectations etc. • Often correctness is more important than fast iteration ML/AI/data science • Workflows are DAGs of arbitrary Python code - the DAG is an end-to-end program by itself, so managing state and data flow within the DAG makes a big difference • Compute needs are demanding: Horizontal scalability, vertical scalability, distributed computing, GPUs eetc • Development is like any software engineering: You need to be able to iterate locally, work with CI/CD etc • Software environment is heterogeneous: you need all kinds of packages written in Python, C, Rust, CUDA drivers etc • Software dependencies are all over the place: There are a zillion libraries (of varying quality) addressing different types of ML/data science/AI use cases • Fast iteration speed is a must - often correctness can be only established by running A/B tests in production

very interesting