Hi, I am facing an issue with loading cards in the UI which is quite similar to this thread but the proposed solution for me does not work. Within my DAG(which I run on AWS Batch) I can successfully view the card locally by running

python flow.py --environment=conda card view end

In the UI I am still getting an error when querying the card via

<https://metaflow-ui.com/api/flows/StagingVisualisationExampleFlow/runs/819/steps/end/tasks/10404/cards>

querying it via CURL returns the following error

Card not found for pathspec StagingVisualisationExampleFlow/819/end/10404

. These are the values for the S3 buckets in the config:

"METAFLOW_DATASTORE_SYSROOT_S3": "<s3://S3_BUCKET/metaflow>"
"METAFLOW_DATATOOLS_SYSROOT_S3": "<s3://S3_BUCKET/data>"

The UI is also deployed on AWS EKS, I have attached a AWS IAM Role which uses AssumeRoleWebIdentity and the following policy document to the AWS EKS Cluster:

data "aws_iam_policy_document" "worker_metaflow_ui_bucket" {
  statement {
    actions = [
      "s3:Get*",
      "s3:List*",
    ]

    resources = [
      "arn:aws:s3:::S3_BUCKET",
      "arn:aws:s3:::S3_BUCKET/*",
    ]
  }
}