I remember hearing Metaflow is experimenting with an OpenLineage integration. Any updates on that? It looks like you can publish custom lineage events to AWS DataZone. It would be so sick to have a Metaflow example that uses • Athena to create some tables in AWS glue catalog • A Metaflow flow that uses Athena and/or duckdb to query the glue catalog to write to a new table with batch inference And then register the Metaflow flow diwnstream of the incoming tables and upstream of the inference table. https://aws.amazon.com/blogs/big-data/amazon-datazone-introduces-openlineage-compatible-data-lineage-visualization-in-preview/

yas ! we have a big release coming up soon related to custom decorators etc. More control over artifacts is the next one up, which can help with this factually you don't have to wait for that feature to land - with the custom decorators you can have

@track_lineage

which pushes the info to your catalog of choice. I can provide an example soon excited

Duuuuude