Hi, I have a very weird problem with Metaflow UI - for runs executed locally I can access task metadata without problems, but for runs executed remotely (on-prem k8s cluster) I got:

Task page encountered an unexpected error. This should not happen and might be caused by unexpected data.

Also it only happens when the first step is executed remotely. When the first step is run locally and the rest of the steps are executed remotely - I can access task metadata without issues. I've checked if data is present in minIO bucket and it is - for all of my runs. Any idea as to what is causing this error?

@plain-breakfast-55936 can you share the versions on metaflow, metaflow-service, and metaflow-ui that you are using? Both locally and on-prem?

I'm not sure what you mean by "both locally and on-prem". I have metaflow python package installed locally, and metaflow-service and metaflow-ui running on-prem so there is only one version for each: metaflow - 2.7.21 metaflow-service - v2.3.3 metaflow-ui - v1.2.4

Thanks - they are not quite the latest versions but recent enough not to be a problem. Do you see any errors in the DevTools

Console

or

Network

tabs of your browser?

start

is the name of the first step:

The error is being caught by the

<ErrorBoundary/>

in

RunPage.tsx

and the error you are seeing in your console screenshot is likely from one of the places in the search screenshot attached. To narrow it down, can you click on

index.tsx:491:36

and share the code around that line?

image.png

Thanks! That's very helpful. I am glad the code obfuscation has cleared up.

Thanks for providing fix! I'll try to apply it to my setup and see if it works.

... I am wondering if your on-prem code is running on a different version of metaflow?

How can I check metaflow version on remote ? I tried printing it with

pip show

and from within metaflow script but without success.

When you are looking at a task/run on MFGUI, you can open a Details section and the version will be shown in the metadata

@brave-lion-15961 thanks! Actually we managed to fix mismatched DAG data by updating image for metadata service in the same way as shown here: https://github.com/outerbounds/metaflow-tools/pull/30/files#diff-2ab5c473a1676709921e61a13a44f909071011bef5ff8ffb04dad464c2f60684R2. Everything works perfectly now.