Happy New Year all,
One of our users has discover...
# ask-metaflowa
acoustic-van-30942
01/04/2024, 5:39 PMHappy New Year all,
One of our users has discovered an error in the Metaflow UI that prevent them from viewing the run parameters due to Internal Error.
Potential bug perhaps? Could someone please kindly help?
✅ 1
acoustic-van-30942
01/04/2024, 5:40 PM Copy code
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/root/services/ui_backend_service/ui_server.py", line 167, in <module>
main()
File "/root/services/ui_backend_service/ui_server.py", line 152, in main
loop.run_forever()
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
self._run_once()
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once
handle._run()
File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
resp = await request_handler(request)
File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_app.py", line 543, in _handle
resp = await handler(request)
File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 114, in impl
return await handler(request)
File "/opt/latest/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 114, in impl
return await handler(request)
File "/root/services/utils/__init__.py", line 89, in wrapper
err_trace = getattr(err, 'traceback_str', None) or get_traceback_str()
File "/root/services/utils/__init__.py", line 84, in wrapper
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/services/ui_backend_service/jupyter/default/metaflow/api/run.py", line 211, in get_run_parameters
combined_results = await self._artifact_store.get_run_parameters(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/services/ui_backend_service/data/cache/store.py", line 204, in get_run_parameters
response = await self.parameter_refiner.postprocess(db_response, invalidate_cache=invalidate_cache)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/services/ui_backend_service/data/refiner/refinery.py", line 78, in postprocess
data = await self.fetch_data(
^^^^^^^^^^^^^^^^^^^^^^
File "/root/services/ui_backend_service/data/refiner/parameter_refiner.py", line 25, in fetch_data
async for event in _res.stream():
File "/root/services/ui_backend_service/data/cache/client/cache_async_client.py", line 106, in wait_iter
raise CacheServerUnreachable()
services.ui_backend_service.data.cache.client.cache_client.CacheServerUnreachablea
ancient-application-36103
01/04/2024, 5:41 PMmeow wave - can you help us with what the actual parameters were?
a
acoustic-van-30942
01/04/2024, 5:42 PMIt's affecting all flows today it seems, and the parameters were a melange: str, list, integers
b
brave-lion-15961
01/04/2024, 5:43 PMCan you share the response to the request that is of the form
/flows/${run.flow_id}/runs/${run.run_number}/parameters👀 1
a
acoustic-van-30942
01/04/2024, 5:50 PMIs this ok?
b
brave-lion-15961
01/04/2024, 5:55 PMThanks - I assume there is nothing of value in the
Previewa
acoustic-van-30942
01/04/2024, 5:56 PMYup that's correct
b
brave-lion-15961
01/04/2024, 5:57 PMJust confirming that the error is happening lower down the stack than the UI
a
acoustic-van-30942
01/04/2024, 6:35 PMSorry Brendan - Are you confirming that, or did you want me to?
b
brave-lion-15961
01/04/2024, 6:38 PMI am confirming that the error is not in the MFGUI code so I can wash my hands and say "Not my problem!" 😉 Don't worry, I'll make sure the folks who know more about the backend service and metaflow itself follow-up with you.
a
acoustic-van-30942
01/04/2024, 6:39 PMSounds good - thanks a bunch!
acoustic-van-30942
01/09/2024, 12:51 AMIssue resolved. Not sure what it was, but probably the service and ui images were using an older version of Metaflow, so just needed to patch
👍🏼 1
👍🏽 1
4 Views