Does metaflow provide a guide/api on cleaning up everything related to a flow run. I am looking to permanently delete: 1. Any rows in the metaflow database referring to a particular runId across all tables 2. Deleting any metadata as well as associated data associated with the flow in the corresponding s3 bucket Both our database & s3 bucket are getting excessively bloated and we are looking to clean up a few things for sanity.

For the datastore, metaflow uses content addressed storage at the flow level to minimize the amount of data stored (objects are not stored redundantly). You should be able to drop all records in the database for a flow name as well as drop the s3 prefix (datastore_root/flow_name) if you want to remove all references to a flow. We don’t ideally recommend this if you have other flows that depend on runs of this flow; but that may or may not be a concern in your case.

I understand that CAS means artifacts may be shared across flow runs. In that case would

artifacts_v3

table serve as source of truth to identify artifacts shared across flow runs by looking at rows with same

sha

across different runs?

Correct

Great, so then we should be reliably able to delete artifacts without any data loss for undeleted flows. Thanks.

Yes - as long as you are careful with the deletes for a flow that might have active runs.

That’s a great idea. Would you be open to perhaps adding a new endpoint to metadata service itself that given a runId executes these delete operations? Or would you like to have that as a standalone script?

Ideally a stand alone option would be better - since these deletes need to be carefully choreographed

Ah, gotcha