hi, any ideas how to clean up storage in .metaflow directory? it saves all the runs data, what if I need to save last N runs data?

data storage across runs in de-duped. given that you have to be careful removing data artifacts for the last N runs. however, metadata for every run (started_at, finished_at timestamps, tags etc.) can be safely removed for any run.

thanks for the answer. still it’s not so obvious for me what are the best practices for this kind of storage optimization. We have a big DS team and all face this problem. Can I kindly ask you to write down some best practices based on your and community experience, which, I believe, is huge.

@gifted-helicopter-98777 can you help me with how you have deployed metaflow, what is the usage pattern and what are you optimizing for (saving storage costs, something else)?

I would also be interested in this, and I think it is a reasonable request since this issue will affect all Metaflow instances that see heavy use over time. IMHO a good starting point would be some documentation about how to safely delete all run information based on a simple retention policy (could be either "last N runs" or "runs older then X days") My org is evaluating Metaflow as a candidate tool right now, and the lack of such a cleanup mechanism is one of the maintenance/ops concerns we have at the moment.

@orange-secretary-61629 - storing more data (metadata and artifacts) with metaflow by design wouldn't necessarily slow the UI or affect the performance of any execution.