Hi all, tried doing some searching to see if someo...
# ask-metaflowb
bulky-portugal-95315
03/22/2023, 3:58 PMHi all, tried doing some searching to see if someone had encountered the same issues we’ve been having with Metaflow UI slowdown. Our metaflow UI has significantly slowed down over time as we have executed more runs on Metaflow and have needed to increase the UI container cpu and memory in order to free up the performance. I’m wondering if we are a unique case in seeing this or we are missing something as far as improving the metaflow UI performance. We only have a few dozen users, but our flow dag’s are fairly complex. Thanks!
✅ 1
a
ancient-application-36103
03/22/2023, 4:26 PM@bulky-portugal-95315 - which views in the UI have slowed down? Can you also comment on what were the original resources for the UI container and what are the newer values?
b
bulky-portugal-95315
03/22/2023, 4:28 PMthe UI main runs view when you open the main UI page at :8083. The original UI container was at 4 vcpu + 30GB mem, we upped to the next level at 8vcpu and 60GB mem. this is deployed on fargate alongside our metadata service in a separate container
a
ancient-application-36103
03/22/2023, 4:37 PMThat should be plenty. How many runs are in-flight concurrently usually? One quick hack while we debug the issue would be to make sure that the run view is pinned to last 30 days (which should happen by default)
b
bulky-portugal-95315
03/22/2023, 4:39 PMwe have anywhere between maybe 20-30 runs running concurrently most times. the L30 is where it ends up default when we go to the UI url at
ip:8083a
ancient-application-36103
03/22/2023, 4:40 PMand how have you deployed the UI? Is it running against a logical replica of the DB or the master DB?
b
bulky-portugal-95315
03/22/2023, 4:40 PMfor a datapoint, i just did a refresh and it takes about 24 sec to load the initial 30 rows
bulky-portugal-95315
03/22/2023, 4:41 PMahhh i believe it’s pointing to the master db actually
a
ancient-application-36103
03/22/2023, 4:41 PMOne scenario where the UI could be slow would be if you haven't set up the indices for the UI
ancient-application-36103
03/22/2023, 4:42 PMWe have UIs that have 0.5M runs with 100s-1000s of concurrent executions that are viewable in the runs view with no visually perceptible delay.
b
bulky-portugal-95315
03/22/2023, 4:43 PMthat’s what i figured, so i’ve been trying to think of what we might be missing. is there a recommended rds instance type as well? I noticed our is only a
m6g.largem6g.12xlargebulky-portugal-95315
03/22/2023, 4:44 PMregarding the indices for the UI, is that documented anywhere respective to metaflow’s data schema?
a
ancient-application-36103
03/22/2023, 4:45 PMYes - here are the details - https://github.com/Netflix/metaflow-service#migration-service
ancient-application-36103
03/22/2023, 4:46 PMThe migration service will apply these migration files - https://github.com/Netflix/metaflow-service/tree/master/services/migration_service/migration_files
b
bulky-portugal-95315
03/22/2023, 4:50 PMgot it, can you help me understand if the rds instance type and/or iops has any impact to the UI?
a
ancient-application-36103
03/23/2023, 6:50 PMit does - the beefier the instance, the faster the queries will perform
👍 1