https://outerbounds.com/ logo
Join Slack
Powered by
Hi Metaflow We had a metaflow installation which w...
# ask-metaflow
i
Hi Metaflow We had a metaflow installation which was working fine until now. Suddenly we are seeing this error and UI is not loading
Copy code
INFO:aiohttp.access:XX.XX.XX.XX [20/Dec/2022:20:59:37 +0000] "GET /api/runs?_order=-ts_epoch&_limit=30&_group_limit=31&_page=1&ts_epoch%3Age=1668902400000 HTTP/1.1" 500 0 "<https://m>*********/?timerange_start=1668902400000" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
ERROR:AsyncPostgresDB:ui:Exception occured
Traceback (most recent call last):
  File "/root/services/ui_backend_service/data/db/tables/base.py", line 228, in execute_sql
    await cur.execute(select_sql, values)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 426, in execute
    await self._conn._poll(waiter, timeout)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 884, in _poll
    raise exc
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 409, in wait_for
    await waiter
concurrent.futures._base.CancelledError
What we can do to recover from this and avoid this in future. Thanks
✅ 1
a
interesting, my hunch would be to check if either DB connection is timing out due to network issues, or queries take too long to execute (maybe db is overloaded)
or maybe too many connections.. though it should be using a fixed size connection pool
that 
CancelledError
if I recall correctly is a bit of a red herring, its from asyncio request being cancelled due to operation timing out, but its not the root cause by itself
i
Workflows are running fine and no issues. This happens only when loading the UI
a
is metadata service and UI point to the same db? or do you use a replica? also if you have access to some performance metrics of the db it may help
asking because UI is running much heavier queries than the metadata service, so in theory its possible that metadata service queries succeed where UI queries take too long
i
Metrics seems not too overload. DB itself is running as a container
DB Metrics
UI Metrics
a
if that CPU usage on the DB is from just metadata service normal operation, i can imagine UI queries could push it way higher
if its easy id try assigning 2-4x more CPU to the db just to see if that helps the problem
(i know 4x is probably excessive, but at least doing this temporarily could help see if overprovisioning CPU helps at all)
i
Sure @narrow-lion-2703 will try and update here if its solves the problem. Thanks
Increasing the DB resources didn’t help (Increased to 4x)
But, noticied something else
UI fails when querying 
/api/runs
if I try that separately, it only fails when 
_order=-ts_epoch
is passed
Copy code
runs?_limit=30&_group_limit=31&_page=1
This works well (both curl and browser)
with 
order
parameter it fails
a
yeah order would make that query much slower
i
Thanks Oleg. Will check
Thanks Oleg, creating that index fixed it
Copy code
CREATE INDEX CONCURRENTLY IF NOT EXISTS runs_v3_idx_epoch_ts_desc ON runs_v3 (ts_epoch DESC);
after this the UI started working and its faster. Thanks 👍💯
👍 1
among us party 2
6 Views
Open in Slack
PreviousNext
Powered by