https://outerbounds.com/ logo
Join Slack
Powered by
Hello all, We are getting the following error in ...
# ask-metaflow
a
Hello all, We are getting the following error in the Metaflow UI. It's preventing users from viewing logs in the UI and logs aren't being propagated to the UI as well. Any insights?
✅ 1
Copy code
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 307, in <module>
    cli(auto_envvar_prefix='MFCACHE')
  File "/opt/latest/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/latest/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/latest/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/latest/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 301, in cli
    Scheduler(store, max_actions).loop()
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 196, in __init__
    self.pool = multiprocessing.Pool(
  File "/usr/local/lib/python3.11/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/usr/local/lib/python3.11/multiprocessing/pool.py", line 215, in __init__
    self._repopulate_pool()
  File "/usr/local/lib/python3.11/multiprocessing/pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/usr/local/lib/python3.11/multiprocessing/pool.py", line 329, in _repopulate_pool_static
    w.start()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/lib/python3.11/multiprocessing/context.py", line 281, in _Popen
    return Popen(process_obj)
  File "/usr/local/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/lib/python3.11/multiprocessing/popen_fork.py", line 71, in _launch
    code = process_obj._bootstrap(parent_sentinel=child_r)
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 29, in execute_action
    execute(tempdir, action_cls, request)
  File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 51, in execute
    res = action_cls.execute(
  File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 133, in execute
    with streamed_errors(stream_output):
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/root/services/ui_backend_service/data/cache/utils.py", line 130, in streamed_errors
    get_traceback_str()
  File "/root/services/ui_backend_service/data/cache/utils.py", line 124, in streamed_errors
    yield
  File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 134, in execute
    task = Task(pathspec, attempt=attempt)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/latest/lib/python3.11/site-packages/metaflow/client/core.py", line 1040, in __init__
    super(Task, self).__init__(*args, **kwargs)
  File "/opt/latest/lib/python3.11/site-packages/metaflow/client/core.py", line 305, in __init__
    self._object = self._get_object(*ids)
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/latest/lib/python3.11/site-packages/metaflow/client/core.py", line 333, in _get_object
    result = self._metaflow.metadata.get_object(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/latest/lib/python3.11/site-packages/metaflow/metadata/metadata.py", line 425, in get_object
    pre_filter = cls._get_object_internal(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/latest/lib/python3.11/site-packages/metaflow/plugins/metadata/service.py", line 242, in _get_object_internal
    cls._supports_attempt_gets = version is not None and LooseVersion(
                                                         ^^^^^^^^^^^^

NameError: name 'LooseVersion' is not defined
b
which version of the UI are you running? This was caused by an issue in the bundled metaflow client with one of the releases, but it has been amended since.
a
Probably an older version. How do we check again? We are building manually from 
<https://github.com/Netflix/metaflow-service>
, so maybe we just need to refresh the fork.
b
the versions are usually visible through the top right menu (on my phone so pictures later if necessary). it should list both the metaflow-service and the UI versions under it
yeh, so the issue was introduced in Metaflow client 
2.10.4
and fixed in 
2.10.5
so if your metaflow-service was built before that release, it bundled the borked version of the client which causes the log issue.
from the metaflow-service releases, 
2.4.4
is affected, but 
2.4.5
should contain the fix. Actually after checking, both releases contain the 2.10.5 client so neither should be affected.
A simple rebuild in your case should be enough as the Metaflow client is loosely pinned (no need to even refresh the fork) Let me know if this gets solved 🙂
a
Thanks Sakari - currently rebuilding. Will keep you posted. This specifically affects the ui backend service and not the front-end right?
b
correct, it is an issue with ui-backend-service, and the version of metaflow client that it bundled during building (2.10.4)
a
Got it! I went through the effort of refreshing the fork too (I didn't see your latest message), and noticed the docker file is now using multi-stages, so I implemented that as well in our security hardened image.
b
for future reference: if you want to verify manually which version of the client was bundled, you can pull the image that was built and do
Copy code
docker run --rm -it --entrypoint bash [image-name]
source /opt/latest/bin/activate
pip show metaflow
to check the version. This is the only version info that the UI doesn't currently expose
👍 1
a
Got it and good point, thanks Sakari!
Ok can confirm it's back up and running and I see the UI has seen some changes/updates. Looks beautiful!
👌 1
b
excellent, glad to hear 🙂
a
Thanks a bunch for your help! 🙂
2 Views
Open in Slack
PreviousNext
Powered by