Hi, I was wondering if there is anyway of "cleaning" the metadata database. We apparently hit our 100GB disk limit. What can/should we do? (vacuum/delete older records... etc)... also what exactly takes the space? [ we increased disk space for now...]

@breezy-butcher-29663 removing rows from the database will make older executions inaccessible by the Metaflow client. Is that a desired behavior?

I'd say its an acceptable behaviour. the data is mainly used for debugging. so for our use case we only need the last year at max. ( on AWS we go from 50$ per month to 100$ per month in moving from 100 to 200 GB with read replica) https://aws.amazon.com/rds/postgresql/pricing/?pg=pr&loc=3

You should be able to drop rows in all tables based on timestamps (it shouldn't result in a table lock). I would recommend taking a backup of the database before you drop anything.

Something to potentially keep an eye on that could lead to significantly more disk usage, depending on your deployment, is having a logical replica (e.g. to isolate the UI service from the core metadata API). If the logical replica DB subscription ever stops receiving updates, the primary DB's replication slot will queue up all the changes without being drained, resulting in a significant bump to disk usage. Anecdotally I've had this happen (breaking the replica slot after performing a DB migration on the primary) and within 3 days it blew up storage by 70GB. Dropping the original replication slot freed up the extra disk usage, brought CPU back to normal, and then just needed to recreate the UI replica. Pic below shows the usage throughout that period, including RDS adding storage, before it being released at the end. Not sure how many people are running with this kind of setup though, so it may not be applicable to most!