Hi Team, My AWS-based Metaflow UI isn't displaying the logs and threw such errors as:

metaflow.plugins.metadata.service.ServiceException: Metadata request (/flows/DataProcessingFlow/runs/sfn-c29232ad-9c64-4f3f-b145-37990414963c/steps/process/tasks/f4a8c286-743b-4b51-b41d-e414a278963b/metadata) failed (code 500): "{\"err_msg\": {\"type\": \"timeout error\"}}"

So I restarted it (both UI and metadata service), but now, it's a different error:

Status: 500
Title: Unknown error
Detail: -
ID: generic-error
Application version: v1.3.12 - a86f9371f74d7ddf89c1ff46b22a332eb5e23fc7 - production
Service version: 2.4.10--

And in the ECS tasks, I see the following error:

ERROR:AsyncPostgresDB:global:Exception occurred metaflow-ui-service

Traceback (most recent call last):  metaflow-ui-service
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 500, in wait_for    metaflow-ui-service
return fut.result() metaflow-ui-service
^^^^^^^^^^^^    metaflow-ui-service
asyncio.exceptions.CancelledError   metaflow-ui-service
The above exception was the direct cause of the following exception:    metaflow-ui-service
Traceback (most recent call last):  metaflow-ui-service
File "/root/services/data/postgres_async_db.py", line 273, in execute_sql   metaflow-ui-service
body, pagination = await _execute_on_cursor(cur)    metaflow-ui-service
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   metaflow-ui-service
File "/root/services/data/postgres_async_db.py", line 238, in _execute_on_cursor    metaflow-ui-service
await _cur.execute(select_sql, values)  metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 426, in execute   metaflow-ui-service
await self._conn._poll(waiter, timeout) metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 884, in _poll metaflow-ui-service
raise exc   metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 881, in _poll metaflow-ui-service
await asyncio.wait_for(self._waiter, timeout)   metaflow-ui-service
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 502, in wait_for    metaflow-ui-service
raise exceptions.TimeoutError() from exc    metaflow-ui-service
TimeoutError    metaflow-ui-service
ERROR:AsyncPostgresDB:global:Exception occurred metaflow-ui-service
Traceback (most recent call last):  metaflow-ui-service
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 500, in wait_for    metaflow-ui-service
return fut.result() metaflow-ui-service
^^^^^^^^^^^^    metaflow-ui-service
asyncio.exceptions.CancelledError   metaflow-ui-service
The above exception was the direct cause of the following exception:    metaflow-ui-service
Traceback (most recent call last):  metaflow-ui-service
File "/root/services/data/postgres_async_db.py", line 273, in execute_sql   metaflow-ui-service
body, pagination = await _execute_on_cursor(cur)    metaflow-ui-service
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   metaflow-ui-service
File "/root/services/data/postgres_async_db.py", line 238, in _execute_on_cursor    metaflow-ui-service
await _cur.execute(select_sql, values)  metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 426, in execute   metaflow-ui-service
await self._conn._poll(waiter, timeout) metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 884, in _poll metaflow-ui-service
raise exc   metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 881, in _poll metaflow-ui-service
await asyncio.wait_for(self._waiter, timeout)   metaflow-ui-service
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 502, in wait_for    metaflow-ui-service
raise exceptions.TimeoutError() from exc    metaflow-ui-service
TimeoutError

Can someone please help?

Hey, I seem to be seeing this error myself too. The RDS instance is a t3.small in AWS. I don't see any cpu or memory issues with the db and also don't see any resource issues. Out of interest what did you upgrade your instance from and too if you can remember?