Hello team,
How do we delete old pipeline runs on Metaflow and remove the data from s3?
✅ 1
s
square-wire-39606
04/24/2025, 7:52 PM
you can scrub the data from the metadata db so that the ui no longer references it. re: data in s3 - you would want to delete the files referenced in the rows deleted in the metadata db that are not referenced in any other rows that are not deleted.
a
acoustic-van-30942
04/24/2025, 7:54 PM
But sounds like there's no easy way for a user to do this through metaflow api commands
s
square-wire-39606
04/24/2025, 7:55 PM
since we use content addressed storage to reduce the overall storage footprint in blob store, there are a bunch of gymnastics folks need to do to clean up the s3 bucket. however, in practice, just leaning on tiered storage offerings usually drops the cost profile significantly
square-wire-39606
04/24/2025, 7:56 PM
we do have work underway to tag assets for lifecycle management, but hard to pin point if/when it will land in oss