https://outerbounds.com/ logo
Join Slack
Powered by
Hi all, I'm encountering an issue in the UI. When...
# ask-metaflow
a
Hi all, I'm encountering an issue in the UI. When I use the DNS domain name as the service url in the Metaflow config, the UI misrepresents the flow as a failure while the flow is in progress. Inspecting the elements, I do see an error 
CardNotPresentException
. The weird thing is that eventually, the UI recalibrates, and depicts the flow correctly. I am not able to recreate this problem when I use the underlying API Gateway endpoint. This momentary issue only occurs when I use the DNS domain name .
👀 1
Here is the stacktrace:
Copy code
{
  "id": "CardNotPresentException",
  "traceback": "Traceback (most recent call last):\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/runpy.py\", line 197, in _run_module_as_main\n    return _run_code(code, main_globals, None,\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/root/services/ui_backend_service/data/cache/client/cache_server.py\", line 307, in <module>\n    cli(auto_envvar_prefix='MFCACHE')\n  File \"/opt/latest/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\n    return self.main(*args, **kwargs)\n  File \"/opt/latest/lib/python3.9/site-packages/click/core.py\", line 1053, in main\n    rv = self.invoke(ctx)\n  File \"/opt/latest/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n  File \"/opt/latest/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\n    return __callback(*args, **kwargs)\n  File \"/root/services/ui_backend_service/data/cache/client/cache_server.py\", line 301, in cli\n    Scheduler(store, max_actions).loop()\n  File \"/root/services/ui_backend_service/data/cache/client/cache_server.py\", line 196, in __init__\n    self.pool = multiprocessing.Pool(\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/context.py\", line 119, in Pool\n    return Pool(processes, initializer, initargs, maxtasksperchild,\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/pool.py\", line 212, in __init__\n    self._repopulate_pool()\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/pool.py\", line 303, in _repopulate_pool\n    return self._repopulate_pool_static(self._ctx, self.Process,\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/pool.py\", line 326, in _repopulate_pool_static\n    w.start()\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/process.py\", line 121, in start\n    self._popen = self._Popen(self)\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/context.py\", line 277, in _Popen\n    return Popen(process_obj)\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/popen_fork.py\", line 19, in __init__\n    self._launch(process_obj)\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/popen_fork.py\", line 71, in _launch\n    code = process_obj._bootstrap(parent_sentinel=child_r)\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/process.py\", line 315, in _bootstrap\n    self.run()\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/process.py\", line 108, in run\n    self._target(*self._args, **self._kwargs)\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/multiprocessing/pool.py\", line 125, in worker\n    result = (True, func(*args, **kwds))\n  File \"/root/services/ui_backend_service/data/cache/client/cache_worker.py\", line 29, in execute_action\n    execute(tempdir, action_cls, request)\n  File \"/root/services/ui_backend_service/data/cache/client/cache_worker.py\", line 51, in execute\n    res = action_cls.execute(\n  File \"/root/services/ui_backend_service/data/cache/get_data_action.py\", line 116, in execute\n    result = cls.fetch_data(target, stream_output)\n  File \"/root/services/ui_backend_service/data/cache/get_cards_action.py\", line 58, in fetch_data\n    cards = {card.hash: _card_item(card) for card in get_cards(task)}\n  File \"/usr/local/pyenv/versions/latest/lib/python3.9/contextlib.py\", line 137, in __exit__\n    self.gen.throw(typ, value, traceback)\n  File \"/root/services/ui_backend_service/data/cache/utils.py\", line 130, in streamed_errors\n    get_traceback_str()\n  File \"/root/services/ui_backend_service/data/cache/utils.py\", line 124, in streamed_errors\n    yield\n  File \"/root/services/ui_backend_service/data/cache/get_cards_action.py\", line 58, in fetch_data\n    cards = {card.hash: _card_item(card) for card in get_cards(task)}\n  File \"/opt/latest/lib/python3.9/site-packages/metaflow/plugins/cards/card_client.py\", line 260, in get_cards\n    card_paths, card_ds = resolve_paths_from_task(\n  File \"/opt/latest/lib/python3.9/site-packages/metaflow/plugins/cards/card_resolver.py\", line 29, in resolve_paths_from_task\n    card_paths_found = card_datastore.extract_card_paths(\n  File \"/opt/latest/lib/python3.9/site-packages/metaflow/plugins/cards/card_datastore.py\", line 296, in extract_card_paths\n    return self._list_card_paths(\n  File \"/opt/latest/lib/python3.9/site-packages/metaflow/plugins/cards/card_datastore.py\", line 230, in _list_card_paths\n    raise CardNotPresentException(\n\nmetaflow.plugins.cards.exception.CardNotPresentException: Card not found for pathspec TuningFlow/7/start/39\n",
  "detail": "Card not found for pathspec TuningFlow/7/start/39",
  "status": 500,
  "title": "Internal Server Error",
  "type": "about:blank"
}
b
Does your setup include two deployments of 
metaflow-service
?
a
Just 1 deployment.
b
What version do you see with each endpoint? (Click on the 
Quick Links
button in the UI)
a
I don't see a version, which could be because we built this image manually using this https://github.com/Netflix/metaflow-ui/blob/master/Dockerfile and then using hardened base image
b
Do you get anything when you load https://YOUR_DOMAIN/api/version in your browser?
a
Yes - 
2.3.5--
b
and when you switch endpoints, do you get the same?
a
Ok also 
2.3.5
b
That rules out the endpoints pointing to mismatched versions
Since cards are stored in S3, is it possible that your two endpoints are configured with different privileges? And one of them is not able to access S3?
a
I don't think so. I also see the cards in the UI eventually.
b
Are you seeing tasks marked as failed, and then eventually being marked as successful?
a
Yes that's right.
b
OK - the endpoints may not be directly related. It could be that the tasks are failing heartbeat checks. You could try increasing the 
WAIT_TIME
variable https://github.com/Netflix/metaflow-service/blob/master/services/ui_backend_service/docs/environment.md#heartbeat-intervals
a
It doesn't seem to be an issue when I use the underlying API Gateway endpoint. Only when I use the DNS domain. But yes - I can try increasing the 
WAIT_TIME
4 Views
Open in Slack
PreviousNext
Powered by