Hi team 👋 Getting some metaflow_ui errors. The ui loads however the data doesnt and gives generic error 500. Please see screenshot. The setup we have is the ui_backend service is on k8s which is linked to an aws service catalog machine via envoy and then thats linked to my browser with portforwarding. Its forwarded to 127.0.0.1:1234, however it seems the ui wants to load the data from 127.0.0.1:8083 I can get the data to load fine if I forward with nginx locally using this:

server {
    listen 8083;

    location / {
        proxy_pass <http://localhost:1234>;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Is there some environment variable or something I can set somewhere so that the ui tries 1234 instead of 8083? Ive tried setting

METAFLOW_SERVICE_PROXY="<http://127.0.0.1:1234>"

on the k8s that runs the ui backend but no luck. Thanks for the help

Have you tried setting that environment variable for the metaflow-ui service> https://github.com/Netflix/metaflow-ui/blob/4d79394835bf09d7b23fbf1403cfb73178bbe194/docs/README.md?plain=1#L27

Yep I've tried setting that env variable before running the

ui_backend_service

command in k8s deployment

Is it set when you run

yarn start

for the

metaflow-ui

service?

Actually we dont use yarn start we just have a bash script that runs with the command

#!/bin/bash

source /repo/ui_config.env
ui_backend_service

the sourced config file exports that env var

Did you try

REACT_APP_METAFLOW_SERVICE

?

Nope Ill give it a go, set this on the ui_backed service right?

No. The

metaflow-ui

service. This is the service that runs the UI.

We only have metadata_service and ui_backend_service running, ill double check soon though. What repo is the

metaflow-ui

from?

https://github.com/Netflix/metaflow-ui. You definitely have it running as it provides the UI in your first screenshot. I think it's a matter of making sure that env var is available to it when it starts up.

also see this thread https://outerboundsco.slack.com/archives/C02116BBNTU/p1699901478423229?thread_ts=1699888385.819529&cid=C02116BBNTU

Looks like were running the metaflow-ui using this script https://github.com/Netflix/metaflow-service/blob/master/services/ui_backend_service/download_ui.sh This is being done inside the image for the k8s deployment, ive tried setting that env var there but no luck

# Install UI front-end
ENV UI_ENABLED=1
ENV UI_VERSION=v1.3.5
ENV REACT_APP_METAFLOW_SERVICE="<http://127.0.0.1:1234>"
RUN UI_ENABLED=$UI_ENABLED UI_VERSION=$UI_VERSION REACT_APP_METAFLOW_SERVICE=$REACT_APP_METAFLOW_SERVICE /repo/services/ui_backend_service/download_ui.sh

yep take a look at that helm chart https://github.com/outerbounds/metaflow-tools/tree/master/k8s/helm/metaflow, it deploys ui as a separate container in k8s and doesn't use baked-in version in the service image that download_ui.sh downloads

@narrow-lion-2703 bit confused by this, are you saying I should try do the metaflow-ui in a container instead of the download_ui.sh script? So this would be on the same k8s deployment, one container for ui_backend_service and one for metaflow_ui

yes, separate container for the UI bits. There are many ways to deploy this whole thing, but the one with separate container is how most people do it (ignoring download_ui.sh)

Hi @narrow-lion-2703 could I ask for your help again please? I've got the ui_frontend and ui_backend running in containers inside a k8s deployment however when I load the ui it looks like the backend is still looking for the static downloaded files from the download script. Error from loading ui:

[Errno 2] No such file or directory: '/repo/metaflow-service/services/ui_backend_service/ui/index.html'

How do I tell the ui_backend to load from the ui_frontend container? Or should I be hitting the ui_frontend first and then redirecting to the backend for api calls?

The usual setup is you have ingress (or some sort of proxy) that is configured to route /api subpath to the backend and everything else to the frontend container. Your browser would hit ui_frontend first for index html and css/js, and that js code would send request requests to /api/... (from the browser)