Hi all, Had a general question about Metaflow's integration with data catalogs. Looks like we will be leveraging Atlan as our centralized data catalog and it will be used to trace data lineage. Curious on gathering your insight on how we can use Metaflow to interoperate with data catalogs to get full end-to-end data lineage.

great question, a deep topic! 🙂 long story short: One of the key concepts is to make sure that you store a stable pointer to the (version of) data used as an artifact in every run. This way you can trace everything happening inside Metaflow back to your data catalogue / data warehouse

Thanks Ville. Yup that makes sense. We were doing a brainstorming session and arrived at a similar conclusion. We basically need some sort of

uuid

that is specified in the data catalog, and then we need to standardize Metaflow flows such that they capture this

uuid

when pulling in data (perhaps through a custom decorator of sorts)?

yeah, you can have a custom data access library that interacts with your data catalogue and data warehouse. You can implement it as a decorator or a Mixin class as exemplified here. In both of these cases the data access library can store the lineage information as artifacts automatically, guaranteeing that you get it recorded consistently across all projects. As an added benefit, the custom decorator could have a custom

@card

which provides a human-readable report about the data used, making the lineage tracking even easier

Oh wow - that's really cool. I hadn't thought about using cards, but that makes perfect sense! Thanks so much - this is great!

for further inspiration, take a look at this dataset proposal by @cuddly-rocket-69327 https://outerbounds-community.slack.com/archives/C020U025QJK/p1627749955005700

Sure thing 🙏 - will keep that in mind. We are still a ways off from this - more just preparing for the future when we will need some sort of integration :)

Heya both, hope it's ok for me to jump in here! We're having very similar conversations at the moment. Could I ask: what exact lineage relationship you are proposing to capture - is it

CatalogInput

->

Run

->

CatalogOutput

? If so, can you actually have total reproducibility of data? In a development context, we are often running `Flow`s that are not checked into source and so we cannot associated

Run

(and, consequently

CatalogOutput

) with a given SHA. Does that make sense?

Hi Thomas, I think we are looking for

CatalogOutput

->

Run

->

Metadata in Model Registry