The human-centric platform for production ML & AI

Outerbounds

Does MetaFlow keep track of datasets accessed?
e.g. While training, if user accesses a dataset from s3, does MetaFlow track the dataset accessed?

We want to collect model lineage where at the end of a model training, we know what data was used and we can emit that to our lineage tracking systems. MLFlow is one we are using at the moment but it's `autolog` feature is buggy.

I see `MetaFlow.S3` has Built-in support for lineage and versioning. I want to explore that via some examples.

not automatically but you can easily do it by assigning lineage data in `self.` artifacts. For instance you can have a small custom library that does it automatically

if you instantiate `S3` as `metaflow.S3(run=self)` it'll use a versioned prefix for storing data by default but reads you can track through artifacts

I'm very new to MetaFlow. Can you share an example or a tutorial around this?