what is the best way to cleanup the metaflow s3 bucket? my bucket is currently 3.6TB 🤯

congrats 🙂 a few approaches: • artifacts are stored for each flow separately, so you can take an inventory of flows and if they are flows that no on cares about, you can delete the whole prefix for that flow. • if you want to clean up old stuff across flows, you can simply set up a lifecycle policy that deletes old objects. However, this may affect even newer runs that share artifacts through content-addressed storage, so this is risky. • a safer approach would be to enable S3 intelligent tiering and let it move infrequently accessed artifacts to cold storage. If you have objects that are never accessed, you can wipe them out after a while

I've seen similar threads a couple of times now, but not experienced this issue myself yet. One issue I've seen mentioned in this regard is that the flow code can be old and if you set up an S3 lifecycle policy it can randomly delete your flow code. One question from me would be whether it is possible to configure Metaflow to save the code to a separate path to the path used to save data artefacts? I guess that the flow code will be very small in comparison to the data artefacts, so persisting this should not be an issue.

yep, the code package is stored in the same content address storage as artifacts, but in the lifecycle management point of view it might be beneficial to store it elsewhere note that even with a separate prefix for code, deleting old artifacts might still break running flows as artifacts could disappear on the fly, so we want to provide a more principled approach in the long term

note that even with a separate prefix for code, deleting old artifacts might still break running flows as artifacts could disappear on the fly, so we want to provide a more principled approach in the long term

-> Yep, I'm aware of that. But it feels reasonable (to me anyway) to expect people to put long persisting artifacts in a separate place and manage them differently, e.g., with the

metaflow.S3

client and a special prefix.

unfortunately there isn't a straightforward way to change it, since it uses the same code path as the artifacts. Technically you can get the addresses of code packages through Client API, so you could e.g. attach an AWS tag to them and assign a separate lifecycle policy to the tag not exactly a one-click thing but it should work today