Hi, I am facing an issue with loading cards in the UI which is quite similar to this thread but the proposed solution for me does not work. Within my DAG(which I run on AWS Batch) I can successfully view the card locally by running

python flow.py --environment=conda card view end

In the UI I am still getting an error when querying the card via

<https://metaflow-ui.com/api/flows/StagingVisualisationExampleFlow/runs/819/steps/end/tasks/10404/cards>

querying it via CURL returns the following error

Card not found for pathspec StagingVisualisationExampleFlow/819/end/10404

. These are the values for the S3 buckets in the config:

"METAFLOW_DATASTORE_SYSROOT_S3": "<s3://S3_BUCKET/metaflow>"
"METAFLOW_DATATOOLS_SYSROOT_S3": "<s3://S3_BUCKET/data>"

The UI is also deployed on AWS EKS, I have attached a AWS IAM Role which uses AssumeRoleWebIdentity and the following policy document to the AWS EKS Cluster:

data "aws_iam_policy_document" "worker_metaflow_ui_bucket" {
  statement {
    actions = [
      "s3:Get*",
      "s3:List*",
    ]

    resources = [
      "arn:aws:s3:::S3_BUCKET",
      "arn:aws:s3:::S3_BUCKET/*",
    ]
  }
}

Hi @ambitious-bird-15073 . To start with, can you confirm that you can see/ find the card in S3? Then can you share the result of the request to /cards from within your browser? ( rather than curl)

Hey Yudhiesh 👋 can you also check if the Metaflow config file of the flow creating the card and the Metaflow related environment variables in the UI service container are set the same ? Have by any chance also set a METAFLOW_CARD_S3ROOT in either your Metaflow config file or the Metaflow UI service container ?

@brave-lion-15961 yes I can find it at this path

<s3://S3_BUCKET/metaflow/mf.cards/StagingVisualisationExampleFlow/runs/821/tasks/10414/cards/default-aa9a8db9afca1064fdd1bc1ad7c882cf5f8b86cd.html>

Making a GET request to the save URL returns the same response as running it via curl:

"{\"id\": \"CardNotPresentException\", \"traceback\": \"Traceback (most recent call last):\\n  File \\\"/usr/lib/python3.7/runpy.py\\\", line 193, in _run_module_as_main\\n    \\\"__main__\\\", mod_spec)\\n  File \\\"/usr/lib/python3.7/runpy.py\\\", line 85, in _run_code\\n    exec(code, run_globals)\\n  File \\\"/root/services/ui_backend_service/data/cache/client/cache_server.py\\\", line 302, in <module>\\n    cli(auto_envvar_prefix='MFCACHE')\\n  File \\\"/opt/latest/lib/python3.7/site-packages/click/core.py\\\", line 1128, in __call__\\n    return self.main(*args, **kwargs)\\n  File \\\"/opt/latest/lib/python3.7/site-packages/click/core.py\\\", line 1053, in main\\n    rv = self.invoke(ctx)\\n  File \\\"/opt/latest/lib/python3.7/site-packages/click/core.py\\\", line 1395, in invoke\\n    return ctx.invoke(self.callback, **ctx.params)\\n  File \\\"/opt/latest/lib/python3.7/site-packages/click/core.py\\\", line 754, in invoke\\n    return __callback(*args, **kwargs)\\n  File \\\"/root/services/ui_backend_service/data/cache/client/cache_server.py\\\", line 298, in cli\\n    Scheduler(store, max_actions).loop()\\n  File \\\"/root/services/ui_backend_service/data/cache/client/cache_server.py\\\", line 199, in __init__\\n    maxtasksperchild=512,  # Recycle each worker once 512 tasks have been completed\\n  File \\\"/usr/lib/python3.7/multiprocessing/context.py\\\", line 119, in Pool\\n    context=self.get_context())\\n  File \\\"/usr/lib/python3.7/multiprocessing/pool.py\\\", line 176, in __init__\\n    self._repopulate_pool()\\n  File \\\"/usr/lib/python3.7/multiprocessing/pool.py\\\", line 241, in _repopulate_pool\\n    w.start()\\n  File \\\"/usr/lib/python3.7/multiprocessing/process.py\\\", line 112, in start\\n    self._popen = self._Popen(self)\\n  File \\\"/usr/lib/python3.7/multiprocessing/context.py\\\", line 277, in _Popen\\n    return Popen(process_obj)\\n  File \\\"/usr/lib/python3.7/multiprocessing/popen_fork.py\\\", line 20, in __init__\\n    self._launch(process_obj)\\n  File \\\"/usr/lib/python3.7/multiprocessing/popen_fork.py\\\", line 74, in _launch\\n    code = process_obj._bootstrap()\\n  File \\\"/usr/lib/python3.7/multiprocessing/process.py\\\", line 297, in _bootstrap\\n    self.run()\\n  File \\\"/usr/lib/python3.7/multiprocessing/process.py\\\", line 99, in run\\n    self._target(*self._args, **self._kwargs)\\n  File \\\"/usr/lib/python3.7/multiprocessing/pool.py\\\", line 121, in worker\\n    result = (True, func(*args, **kwds))\\n  File \\\"/root/services/ui_backend_service/data/cache/client/cache_worker.py\\\", line 29, in execute_action\\n    execute(tempdir, action_cls, request)\\n  File \\\"/root/services/ui_backend_service/data/cache/client/cache_worker.py\\\", line 56, in execute\\n    invalidate_cache=req.get('invalidate_cache', False))\\n  File \\\"/root/services/ui_backend_service/data/cache/get_data_action.py\\\", line 116, in execute\\n    result = cls.fetch_data(target, stream_output)\\n  File \\\"/root/services/ui_backend_service/data/cache/get_cards_action.py\\\", line 58, in fetch_data\\n    cards = {card.hash: _card_item(card) for card in get_cards(task)}\\n  File \\\"/usr/lib/python3.7/contextlib.py\\\", line 130, in __exit__\\n    self.gen.throw(type, value, traceback)\\n  File \\\"/root/services/ui_backend_service/data/cache/utils.py\\\", line 130, in streamed_errors\\n    get_traceback_str()\\n  File \\\"/root/services/ui_backend_service/data/cache/utils.py\\\", line 124, in streamed_errors\\n    yield\\n  File \\\"/root/services/ui_backend_service/data/cache/get_cards_action.py\\\", line 58, in fetch_data\\n    cards = {card.hash: _card_item(card) for card in get_cards(task)}\\n  File \\\"/opt/latest/lib/python3.7/site-packages/metaflow/plugins/cards/card_client.py\\\", line 207, in get_cards\\n    _get_flow_datastore(task), pathspec=task.pathspec, type=type, card_id=card_id\\n  File \\\"/opt/latest/lib/python3.7/site-packages/metaflow/plugins/cards/card_resolver.py\\\", line 30, in resolve_paths_from_task\\n    card_type=type, card_hash=hash, card_id=card_id\\n  File \\\"/opt/latest/lib/python3.7/site-packages/metaflow/plugins/cards/card_datastore.py\\\", line 223, in extract_card_paths\\n    card_type=card_type, card_hash=card_hash, card_id=card_id\\n  File \\\"/opt/latest/lib/python3.7/site-packages/metaflow/plugins/cards/card_datastore.py\\\", line 164, in _list_card_paths\\n    card_type=card_type,\\n\\nmetaflow.plugins.cards.exception.CardNotPresentException: Card not found for pathspec StagingVisualisationExampleFlow/821/end/10414\\n\", \"detail\": \"Card not found for pathspec StagingVisualisationExampleFlow/821/end/10414\", \"status\": 500, \"title\": \"Internal Server Error\", \"type\": \"about:blank\"}"

Oh. One more question, are you able to see any cards on the UI? I think the glitch is that your

MF_DATASTORE_ROOT

and

METAFLOW_DATASTORE_SYSROOT_S3

are different from your config. Ideally they should be the same in your ui service container as they are in your config ie the value should be

<s3://S3BUCKET/metaflow>

@hallowed-glass-14538 no I do not see any. Great will try it out and update you soon.

What is your metaflow version ?

Do you mean the ui? @hallowed-glass-14538

Oh no no. I mean the metaflow library

There are Flows with 2.7.10, 2.7.11 and 2.7.14

If its deployed like a container, can you bounce the container from the cluster?

Hey sorry @hallowed-glass-14538 I asked my devops guy to try and restart the pod. Waiting for him to do so.

cc. @brief-kite-90012