Hi Everyone, We are facing issues with Metaflow installed on K8s (Metaflow Service, UI and S3 Datastore), where the workflow itself runs successfully in Argo Workflow, but not showing anywhere in the Metaflow UI (no entry on the the database

runs_v3

table as well) Installed via Helm: https://github.com/outerbounds/metaflow-tools/tree/master/k8s/helm/metaflow but with latest version tags of metaflow_service (2.3.3) and ui (1.1.4).

{
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "METAFLOW_DATASTORE_SYSROOT_S3": "s3://****************",
  "METAFLOW_KUBERNETES_NAMESPACE": "metaflow",
  "METAFLOW_KUBERNETES_SERVICE_ACCOUNT": "metaflow-metaflow-service",
  "METAFLOW_SERVICE_URL": "<http://metaflow-metaflow-ui.metaflow.svc.cluster.local:8083/api/metadata>"
}

Thanks

can you also add

METAFLOW_DEFAULT_METADATA

as

service

in your config and try again?

With that added, I’m seeing a different error on the tasks (Tasks are failing). Still not showing up in the metaflow UI

Code package downloaded.
Task is starting.
    Metaflow service error:
    Metadata request (/flows/AlphaFlow) failed (code 405): 405: Method Not Allowed

time="2022-12-18T19:58:30.913Z" level=info msg="no need to save parameter - on overlapping volume: /mnt/out/task_id" argo=true
Error: exit status 1

405 Method Not allowed.

that variable is necessary for the metadata service to record the metadata so that the UI can correctly display the information. When you use

2.2.4

/

v1.0.1

with that variable - does everything work well?

I had similar issues recently and had to set my METAFLOW_KUBERNETES_NAMESPACE to "argo", maybe related

I needed that variable even with

2.2.4

/

1.0.1

@ancient-application-36103. But, it worked and showed up in the UI. You were exactly on point about the variable. My

METAFLOW_KUBERNETES_NAMESPACE

is

metaflow

and Argo is deployed in

metaflow

namespace.

how are you cycling between

2.2.4

and

2.3.3

version of the service?

Helm upgrade with new values for tags (DB is reset on each try - as this was a new install)

@icy-exabyte-25104 is this issue also resolved alongside the other issue?

No, these are 2 different installs @ancient-application-36103. The UI issue is where, we use if for all our development. This issue is a separate install where we are testing our upgrade. If this works, we will upgrade the real environment

Are you able to use the Metaflow Client to list flows that execute on Argo Workflows? Can you share the output of

python flow.py argo-workflows create --only-json

for the deployment where the execution is visible in the Metaflow UI and the deployment where the execution isn't visible?

Sorry Savin, missed your message. Will share the details asap

@icy-exabyte-25104 did you get to the bottom of this? I am having similar issues here.