Hey team, we're currently facing a production outage where the

ui-backend-service

is crashing in loop. It all started yesterday out of the blue, and we thought we had fixed it by stabilizing other parts of the deployments (namely argo workflows related). But this morning, no matter what we have tried (adding requests/limits, replicas, etc) works for us. This exact version 2.5.0 of the image has been running fine with 1 replica since April of 2025, there were no changes to our infra or services 🤯 . this is what we are seeing there. the pods start, the service starts, connects to the db, we start seeing a few "normal requests" then we see the CPU getting pegged to 1 vcpu (single thread?), then after a minute or so (our probes) it just kills it and it comes back again. this is the error message that happens right before the pod gets rescheduled:

2026-01-09T15:40:59.899Z metaflow-ui-backend-service | INFO:aiohttp.access:35.191.238.164 [09/Jan/2026:15:40:59 +0000] "GET /api/ping HTTP/1.1" 200 0 "-" "GoogleHC/1.0"
2026-01-09T15:40:59.899Z metaflow-ui-backend-service | INFO:aiohttp.access:35.191.243.6 [09/Jan/2026:15:40:59 +0000] "GET /api/ping HTTP/1.1" 200 0 "-" "GoogleHC/1.0"
2026-01-09T15:40:59.899Z metaflow-ui-backend-service | INFO:aiohttp.access:35.191.238.163 [09/Jan/2026:15:40:59 +0000] "GET /api/ping HTTP/1.1" 200 0 "-" "GoogleHC/1.0"
2026-01-09T15:40:59.901Z metaflow-ui-backend-service | ERROR:aiohttp.server:Error handling request from 35.191.82.178
2026-01-09T15:40:59.901Z metaflow-ui-backend-service | Traceback (most recent call last):
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 480, in _handle_request
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     resp = await request_handler(request)
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_app.py", line 569, in _handle
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     return await handler(request)
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |            ^^^^^^^^^^^^^^^^^^^^^^
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     return await handler(request)
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |            ^^^^^^^^^^^^^^^^^^^^^^
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     return await handler(request)
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |            ^^^^^^^^^^^^^^^^^^^^^^
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/root/services/ui_backend_service/api/ws.py", line 179, in websocket_handler
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     await ws.prepare(request)
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_ws.py", line 215, in prepare
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     payload_writer = await super().prepare(request)
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_response.py", line 453, in prepare
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     return await self._start(request)
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_response.py", line 461, in _start
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     await self._write_headers()
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_response.py", line 538, in _write_headers
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     await writer.write_headers(status_line, self._headers)
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/opt/latest/lib/python3.11/site-packages/aiohttp/http_writer.py", line 167, in write_headers
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     self._write(buf)
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |   File "/opt/latest/lib/python3.11/site-packages/aiohttp/http_writer.py", line 93, in _write
2026-01-09T15:40:59.901Z metaflow-ui-backend-service |     raise ClientConnectionResetError("Cannot write to closing transport")
2026-01-09T15:40:59.901Z metaflow-ui-backend-service | aiohttp.client_exceptions.ClientConnectionResetError: Cannot write to closing transport

Did you confirm the containers are not getting OOMKilled? I had a issue at one point where some jobs were generating very large log files, users were looking at them in the ui, causing memory usage to go beyond the limit and triggering a crash loop. At first it looked similar to what you described (a normal deployment suddenly failing with no changes to the infra)

yeah, it's entirely CPU related, we are currently running 20 replicas, and made our probes way more forgiving and it's "more stable" but we have nothing that can explain how it suddenly started behaving that way

@ancient-application-36103 can you get someone on your end to take a look?

hi! hard to know what's happening without understanding your setup in detail. unfortunately, the team is a bit bogged down this week - happy to find time next week.

Hello, I would like to see if we can revisit this? Do you have time to help diagnose this and help resolve, we also have a request for a FE edit.