Double-checking/asking to see if there’s a safe way for us to purge older flow information from the Metaflow databases. We are running a dedicated

db.m5.2xlarge

AWS RDS instance to serve a logical replica of

metaflowdb

tables to the Metaflow UI, but even we have issues with (say) a request for a DAG hanging indefinitely with a spinner. Looking at RDS performance insights, we observe the number of active database sessions climbing even with just the single user asking for the DAG.

A

psql

estimate suggests that we have nearly 200m rows in

artifacts_v3

spanning over two years’ worth of Metaflow runs. For compliance reasons we expire Metaflow artifacts from S3 after 30 days, so I imagine holding run info much longer than 30 days doesn’t help us much. FYI: @User @User @User

You should be able to safely drop rows from tables in your db. That shouldn’t have any impact on metaflow if you are already expiring the associated data in s3.

@square-wire-39606 How does Metaflow create the values from

epoch_ts

in

artifact_v3

? Is that

time.time_ns()

?

.

time.time()