Hi Team My AWS based Metaflow UI isn t displaying the logs a Outerbounds #ask-metaflow

Hi Team, My AWS-based Metaflow UI isn't displayin...

echoing-florist-98081

05/16/2024, 8:46 AM

Hi Team, My AWS-based Metaflow UI isn't displaying the logs and threw such errors as:

Copy code

metaflow.plugins.metadata.service.ServiceException: Metadata request (/flows/DataProcessingFlow/runs/sfn-c29232ad-9c64-4f3f-b145-37990414963c/steps/process/tasks/f4a8c286-743b-4b51-b41d-e414a278963b/metadata) failed (code 500): "{\"err_msg\": {\"type\": \"timeout error\"}}"

So I restarted it (both UI and metadata service), but now, it's a different error:

Copy code

Status: 500
Title: Unknown error
Detail: -
ID: generic-error
Application version: v1.3.12 - a86f9371f74d7ddf89c1ff46b22a332eb5e23fc7 - production
Service version: 2.4.10--

And in the ECS tasks, I see the following error:

Copy code

ERROR:AsyncPostgresDB:global:Exception occurred metaflow-ui-service

Traceback (most recent call last):  metaflow-ui-service
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 500, in wait_for    metaflow-ui-service
return fut.result() metaflow-ui-service
^^^^^^^^^^^^    metaflow-ui-service
asyncio.exceptions.CancelledError   metaflow-ui-service
The above exception was the direct cause of the following exception:    metaflow-ui-service
Traceback (most recent call last):  metaflow-ui-service
File "/root/services/data/postgres_async_db.py", line 273, in execute_sql   metaflow-ui-service
body, pagination = await _execute_on_cursor(cur)    metaflow-ui-service
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   metaflow-ui-service
File "/root/services/data/postgres_async_db.py", line 238, in _execute_on_cursor    metaflow-ui-service
await _cur.execute(select_sql, values)  metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 426, in execute   metaflow-ui-service
await self._conn._poll(waiter, timeout) metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 884, in _poll metaflow-ui-service
raise exc   metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 881, in _poll metaflow-ui-service
await asyncio.wait_for(self._waiter, timeout)   metaflow-ui-service
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 502, in wait_for    metaflow-ui-service
raise exceptions.TimeoutError() from exc    metaflow-ui-service
TimeoutError    metaflow-ui-service
ERROR:AsyncPostgresDB:global:Exception occurred metaflow-ui-service
Traceback (most recent call last):  metaflow-ui-service
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 500, in wait_for    metaflow-ui-service
return fut.result() metaflow-ui-service
^^^^^^^^^^^^    metaflow-ui-service
asyncio.exceptions.CancelledError   metaflow-ui-service
The above exception was the direct cause of the following exception:    metaflow-ui-service
Traceback (most recent call last):  metaflow-ui-service
File "/root/services/data/postgres_async_db.py", line 273, in execute_sql   metaflow-ui-service
body, pagination = await _execute_on_cursor(cur)    metaflow-ui-service
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   metaflow-ui-service
File "/root/services/data/postgres_async_db.py", line 238, in _execute_on_cursor    metaflow-ui-service
await _cur.execute(select_sql, values)  metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 426, in execute   metaflow-ui-service
await self._conn._poll(waiter, timeout) metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 884, in _poll metaflow-ui-service
raise exc   metaflow-ui-service
File "/opt/latest/lib/python3.11/site-packages/aiopg/connection.py", line 881, in _poll metaflow-ui-service
await asyncio.wait_for(self._waiter, timeout)   metaflow-ui-service
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 502, in wait_for    metaflow-ui-service
raise exceptions.TimeoutError() from exc    metaflow-ui-service
TimeoutError

Can someone please help?

✅ 1

echoing-florist-98081

05/16/2024, 8:49 AM

Now it throws this..same timeout error from metadata service, I guess:

Copy code

Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 364, in <module> cli(auto_envvar_prefix='MFCACHE') File "/opt/latest/lib/python3.11/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/opt/latest/lib/python3.11/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/opt/latest/lib/python3.11/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/latest/lib/python3.11/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 358, in cli Scheduler(store, max_actions).loop() File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 204, in __init__ self.pool = multiprocessing.Pool( File "/usr/local/lib/python3.11/multiprocessing/context.py", line 119, in Pool return Pool(processes, initializer, initargs, maxtasksperchild, File "/usr/local/lib/python3.11/multiprocessing/pool.py", line 215, in __init__ self._repopulate_pool() File "/usr/local/lib/python3.11/multiprocessing/pool.py", line 306, in _repopulate_pool return self._repopulate_pool_static(self._ctx, self.Process, File "/usr/local/lib/python3.11/multiprocessing/pool.py", line 329, in _repopulate_pool_static w.start() File "/usr/local/lib/python3.11/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/usr/local/lib/python3.11/multiprocessing/context.py", line 281, in _Popen return Popen(process_obj) File "/usr/local/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__ self._launch(process_obj) File "/usr/local/lib/python3.11/multiprocessing/popen_fork.py", line 71, in _launch code = process_obj._bootstrap(parent_sentinel=child_r) File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.11/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 29, in execute_action execute(tempdir, action_cls, request) File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 51, in execute res = action_cls.execute( File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 133, in execute with streamed_errors(stream_output): File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__ self.gen.throw(typ, value, traceback) File "/root/services/ui_backend_service/data/cache/utils.py", line 130, in streamed_errors get_traceback_str() File "/root/services/ui_backend_service/data/cache/utils.py", line 124, in streamed_errors yield File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 136, in execute current_hash = log_provider.get_log_hash(task, logtype) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 295, in get_log_hash return get_log_size(task, logtype) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 196, in get_log_size return task.stderr_size if logtype == STDERR else task.stdout_size ^^^^^^^^^^^^^^^^ File "/opt/latest/lib/python3.11/site-packages/metaflow/client/core.py", line 1330, in stdout_size return self._get_logsize("stdout") ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/latest/lib/python3.11/site-packages/metaflow/client/core.py", line 1446, in _get_logsize meta_dict = self.metadata_dict ^^^^^^^^^^^^^^^^^^ File "/opt/latest/lib/python3.11/site-packages/metaflow/client/core.py", line 1148, in metadata_dict m.name: m.value for m in sorted(self.metadata, key=lambda m: m.created_at) ^^^^^^^^^^^^^ File "/opt/latest/lib/python3.11/site-packages/metaflow/client/core.py", line 1072, in metadata all_metadata = self._metaflow.metadata.get_object( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/latest/lib/python3.11/site-packages/metaflow/metadata/metadata.py", line 425, in get_object pre_filter = cls._get_object_internal( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/latest/lib/python3.11/site-packages/metaflow/plugins/metadata/service.py", line 280, in _get_object_internal v, _ = cls._request(None, url, "GET") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/latest/lib/python3.11/site-packages/metaflow/plugins/metadata/service.py", line 471, in _request raise ServiceException( metaflow.plugins.metadata.service.ServiceException: Metadata request (/flows/CookiePerQueryFlow/runs/sfn-9e03e9a7-0583-6522-ef4f-5f7f01ca5860_6e38b15d-28b5-82de-9d40-69a3792bdf10/steps/end/tasks/c3a43d78-c86c-412b-89e4-2552df9531a7/metadata) failed (code 500): "{\"err_msg\": {\"type\": \"timeout error\"}}"

fast-vr-44972

05/16/2024, 8:57 AM

It looks like something is wrong with the postgres DB used by metadata service. Can you check the RDS and see if it looks healthy?

echoing-florist-98081

05/16/2024, 8:58 AM

The RDS is working fine...it's healthy... i think it's about 14 GB large right now...is it too big to query? if so, can we cleanup the older logs from it?

echoing-florist-98081

05/16/2024, 9:01 AM

I see that the ReadIOPS and Read Latency of the RDS has shot up in the last few days...not sure why

fast-vr-44972

05/16/2024, 9:02 AM

by the way which metaflow metadata version are you using?

echoing-florist-98081

05/16/2024, 9:02 AM

the latest one:

Copy code

"image": "netflixoss/metaflow_metadata_service",

fast-vr-44972

05/16/2024, 9:04 AM

I don't see the version in what you have posted? We use terraform and this is what we are using

Copy code

ui_backend_container_image         = "public.ecr.aws/outerbounds/metaflow_metadata_service:2.4.9"
  ui_static_container_image          = "public.ecr.aws/outerbounds/metaflow_ui:1.3.11"

echoing-florist-98081

05/16/2024, 9:04 AM

without the version lock, it will always pick the latest (default)

fast-vr-44972

05/16/2024, 9:06 AM

Won't it depend on when you last deployed your infra? latest at that time won't be latest right now? By the way I don't think this might be the issue. It's just there were some metadata service query performance improvement

echoing-florist-98081

05/16/2024, 9:07 AM

I restarted the task (redeployed) an hour ago... Yeah, I don't think that this svc itself is an issue... Maybe i will look into the logs of DB Do you know if we can cleanup some logs?

echoing-florist-98081

05/16/2024, 9:08 AM

BTW the latest that I'm using is ui: v1.3.12 svc: 2.4.10--

✅ 1

fast-vr-44972

05/16/2024, 9:09 AM

I think you should be able to. It only stores the flow run metadata. So as long as you don't care about past runs, you should be able to delete it

echoing-florist-98081

05/16/2024, 10:33 AM

Reboot of RDS didn't help... Upgraded the RDS and now it's solved...

fast-vr-44972

05/16/2024, 10:37 AM

Upgraded the version of RDS from what to what?

echoing-florist-98081

05/16/2024, 10:37 AM

RDS instance type, i meant..slight upgrade

👍 1

millions-barista-45672

10/02/2025, 9:14 AM

Hey, I seem to be seeing this error myself too. The RDS instance is a t3.small in AWS. I don't see any cpu or memory issues with the db and also don't see any resource issues. Out of interest what did you upgrade your instance from and too if you can remember?

45 Views

Open in Slack

Previous Next