Does anyone have any advice on tuning the metaflow ui backend+db to decrease the amount of time it takes to load the UI? Specifically, we're running into issues loading stdout/stderr for a given task within the UI. We have logical replication set up such that we have a (primary) metadata DB and a (secondary) UI db, which is the DB that the UI interacts with.

do other views load quickly besides the task view that show `stdout`/`stderr`?

regarding db setup - here are a few notes - https://tide-windshield-1f3.notion.site/Metaflow-DB-Logical-Replica-c08d89debf51410a8d7ae599196d8a33

@victorious-lawyer-58417 yes for the most part. @square-wire-39606 thanks, I'll take a look.

if it's stdout/stderr mainly, it may be more of a caching/service issue rather than the database. If you have cloudwatch setup for the service instance, you can take a look at the overall load/CPU/disk/memory consumption on the instance. If it's too high, upgrading to a larger instance might help

thank you @victorious-lawyer-58417! I'll keep you posted.

hi @victorious-lawyer-58417! I was hoping to get a better idea of what your UI deployment looks like. A few questions: 1. What size development group are you supporting with your UI? We have an active user base of around 15 developers (they don’t all use it at the same time) and are using a

db.m5.2xlarge

RDS instance, but still experiencing some slowness especially when retrieving the logs. 2. Are you using RDS for postgres or aurora? If you’re using RDS, what kind of db instance are you using? 3. Do you know if there’s a safe way to prune the size of the DB (we are concerned mainly because we have logical replication enabled and don’t want to create DB inconsistencies) 4. I looked at the cloudwatch monitoring tab after tinkering with the UI (I don't believe anyone else was hitting it at the same time) and did see a major spike in CPU utilization (screenshot attached). I wanted to know if this looks normal to you. Spiking to 40% CPU utilization with just me hitting the UI looking at the logs for a single flow feels high, no?

I bumped the RDS instance size to

db.m5.4xlarge

but I'm still seeing really high CPU utilization (basically no change from when the instance size was

db.m5.2xlarge

). @victorious-lawyer-58417 I'm wondering if this is more of a caching issue as you mentioned before. any thoughts?

@User @dry-beach-38304 can you take a look at Ishana's questions and chime in too (knowing that your backend is slightly different but the UI is the same). Netflix runs surely the largest deployment of the UI as of today 🙂

chiming in here

What size development group are you supporting with your UI? We have an active user base of around 15 developers (they don’t all use it at the same time) and are using a

db.m5.2xlarge

RDS instance, but still experiencing some slowness especially when retrieving the logs.

At Netflix we have a fairly large group of people using the UI. I cant give you an exact number but hundreds of developers 🙂

Are you using RDS for postgres or aurora? If you’re using RDS, what kind of db instance are you using?

We used to use RDS but recently switched to Aurora Postgres. Mainly to use the horizontal read scaling feature of Aurora. We tweaked the UI backend to separate reads and writes to different Aurora endpoints.

Do you know if there’s a safe way to prune the size of the DB (we are concerned mainly because we have logical replication enabled and don’t want to create DB inconsistencies)

How large is the size of your DB? I dont think a large DB size is causing your issue. Our DB is in the order of several TB for context. In general, I would not recommend pruning just your logical replica. You should be able to get good performance with the right set of indices

I looked at the cloudwatch monitoring tab after tinkering with the UI (I don’t believe anyone else was hitting it at the same time) and did see a major spike in CPU utilization (screenshot attached). I wanted to know if this looks normal to you. Spiking to 40% CPU utilization with just me hitting the UI looking at the logs for a single flow feels high, no?

It does feel high. to clarify, this is on the UI backend service or on the DB? Is the DB CPU also high? If yes, then this usually happens when you are running expensive queries that are accessing disk instead of an index. You can enable RDS performance insights on your AWS RDS instance to find the expensive queries and tune them appropriately.

can you elaborate on these "tweaks to the ui backend to separate reads and writes"? Is it related to this PR? https://github.com/Netflix/metaflow-service/pull/344

yes, we are already using that logic internally and we plan to merge that OSS PR soon