https://outerbounds.com/ logo
Join Slack
Powered by
Hi, I hope this is the right channel for such ques...
# ask-metaflow
s
Hi, I hope this is the right channel for such question. I've deployed metaflow using helm chart to y k8s cluster (using https://github.com/outerbounds/metaflow-tools/tree/master/charts/metaflow). I connected it to my postgresql db, this parts seems to work fine. I have 3 services running, metaflow-service, -ui and -ui-static. So far so good. I've set 
METAFLOW_DEFAULT_DATASTORE=local
as I did not find out how to set up credentials for s3/minio. Now, I port-forward the service (port 8080) to localhost and with 
METAFLOW_SERVICE_URL=<http://localhost:8080/>
I'm able to run example flow; from the logs on service pod it seems it is connecting to remote server which is nice. I can even check the flow status on port 8083, so it seems this is logged corerctly OK, but now I cannot see anything in the UI. The UI (running through ingress) just shows 
Error loading data
on empty dashboard page, nothing about my flow and nothing more specific about the error, just 
Unknown error
/ 
generic-error
. I assume some part of my stack is misconfigured but I can't see any hints anywhere ... Is it complaining about connection to backend? Or is it backend complaning about actual access to datastore? I see nothing relevant in the logs...
a
@hallowed-glass-14538 or @hundreds-zebra-57629 any ideas here?
h
Hey Jiri, have you configured the metaflow UI backend to connect to your database? Also, have you configured the metaflow UI frontend to point to the backed endpoint? Btw, the setup here is a great example on how to get metaflow up and running locally or on-premise.
s
Well, I'm using the helm chart (might look at the dev script later, but in general I cannot direclty create resources in the cluster). As for DB connection, I believe it's here: https://github.com/outerbounds/metaflow-tools/blob/master/charts/metaflow/charts/metaflow-ui/values.yaml#L50-L54 And I have
Copy code
metadatadb:
      host: postgresql.caic
That's the same value as for 
metaflow-service
which is running the initial DB migration. I believe the pods would not be running if the database connection were not configured correctly
UI frontend to backend endpoint: I believe it is this https://github.com/outerbounds/metaflow-tools/blob/master/charts/metaflow/charts/metaflow-ui/values.yaml#L81 And I have in my values:
Copy code
uiStatic:
    metaflowUIBackendURL: <http://metaflow-metaflow-ui:8083/api>
My services:
Copy code
metaflow-metaflow-service                                ClusterIP   10.17.66.169   <none>        8080/TCP,8082/TCP   40h
metaflow-metaflow-ui                                     ClusterIP   10.17.71.199   <none>        8083/TCP            40h
metaflow-metaflow-ui-static                              ClusterIP   10.17.65.12    <none>        3000/TCP            40h
All three pods are running as well and do not report any connection problems
h
If you are accessing the UI by port forwarding the front end pod, then you also need to separately port forward the backed pod and configure the frontend to point to hit the port forwarded backend using local host. In short both front end backend need to be exposed on your local host and the frontend needs to know the local host address of your backend.
s
I'm not accessing it using localhost, I am not port-forwarding, I have ingress configured (there's a section in helm chart for both service and ui)
Where do you mean I should configure the frontent to hit the backend other than 
uiStatic.metaflowUIBackendURL
?
Is it possible to turn on some more debugging so I could know what does this mean exactly?
btw this is green:
h
Do you have ingress exposed for both the frontend/static and the backend components of the Metaflow UI? Can you confirm you can connect to the the backend directly from your local workstation? If so, 
uiStatic.metaflowUIBackendURL
should be set to the URL you are using to hit the backend from your local workstation. For example if the fronted is exposed on the url 
<https://fronted.metaflow.example.com>
and the backend is exposed on the url 
<https://backend.metaflow.example.com>
then set 
uiStatic.metaflowUIBackendURL=<https://backend.metaflow.example.com>
.
s
Yes, I can connect to the backend, at least that I understand that I can do: 1. METAFLOW_SERVICE_URL= < backend service > This works both for localhost:8080 when I port-forward metaflow-metaflow-service running on 8080 and for URL exposed by ingress for this service 2. Then I run 
python example_flow.py run
and it does communicate to remote service, I can even check the service running on 8083 port and the path like /api/metadata/flow/flow-name, this shows some information
Copy code
~/metaflow$ python example_flow.py run
Metaflow 2.15.18 executing ComplexFlowExample for user:jirisuchomel
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2025-06-25 14:00:26.562 Workflow starting (run-id 3):
2025-06-25 14:00:26.951 [3/start/14 (pid 72598)] Task is starting.
2025-06-25 14:00:27.862 [3/start/14 (pid 72598)] Starting the complex flow example!
2025-06-25 14:00:28.047 [3/start/14 (pid 72598)] Task finished successfully.
2025-06-25 14:00:28.179 [3/load_data/15 (pid 72602)] Task is starting.
2025-06-25 14:00:29.066 [3/load_data/15 (pid 72602)] Starting the data load!
2025-06-25 14:00:29.069 [3/load_data/15 (pid 72602)] Preprocessing data
2025-06-25 14:00:29.490 [3/load_data/15 (pid 72602)] Task finished successfully.
2025-06-25 14:00:29.633 [3/train_model/16 (pid 72608)] Task is starting.
2025-06-25 14:00:31.289 [3/train_model/16 (pid 72608)] Model trained with accuracy: 0.1
2025-06-25 14:00:31.290 [3/train_model/16 (pid 72608)] Predictions: [2 4 1 2 3 1 5 7 1 5 4 3 1 3 1 1 5 4 6 7 1 9 3 4 3 5 7 5 9 6 6 5 1 1 0 3 6
2025-06-25 14:00:31.536 [3/train_model/16 (pid 72608)] 3 6 2 5 7 1 4 3 6 9 6 4 6 3 5 1 1 3 5 3 0 4 4 0 1 7 3 2 1 4 1 4 6 5 3 1 1
2025-06-25 14:00:31.536 [3/train_model/16 (pid 72608)] 0 1 5 9 5 2 1 1 3 3 3 6 3 9 5 6 4 4 3 1 4 2 9 1 1 6]
2025-06-25 14:00:31.537 [3/train_model/16 (pid 72608)] Task finished successfully.
2025-06-25 14:00:31.672 [3/predict/17 (pid 72612)] Task is starting.
2025-06-25 14:00:32.628 [3/predict/17 (pid 72612)] Predictions: [8 8 8 8 8 8 8 8 8 8]
2025-06-25 14:00:32.860 [3/predict/17 (pid 72612)] Task finished successfully.
2025-06-25 14:00:32.989 [3/end/18 (pid 72616)] Task is starting.
2025-06-25 14:00:33.881 [3/end/18 (pid 72616)] Flow is complete!
2025-06-25 14:00:34.076 [3/end/18 (pid 72616)] Task finished successfully.
2025-06-25 14:00:34.077 Done!
And my 
uiStatic.metaflowUIBackendURL
is set to http://metaflow-metaflow-ui:8083/api , i.e. to backend service, is that correct? I don't think this should point to the service on 8080
metaflow-ui.uiBackend.metaflowServiceURL
is http://metaflow-metaflow-ui:8083/api/metadata
it's interesting none of the url points to the service running on 8080 though
h
uiStatic.metaflowUIBackendURL
is set to http://metaflow-metaflow-ui:8083/api , i.e. to backend service,
I believe this is the main issue. It should be set to the public endpoint and not the internal k8s service endpoint. When browsing the Metaflow UI, your connection does NOT go [browser] -> [Frontend ] -> [backend]. It goes [browser] -> [Frontend (returns a url for the backend based on its config)] -> [browser] -> [backend].
👀 1
s
Yes! Thanks that finally worked!
Thanks a lot @hundreds-zebra-57629; during the deployment I made few changes to the chart, I'll offer them as PR.
h
Thank you! I have added a couple of comments, but looks good for the most part
s
I amended my commits
3 Views
Open in Slack
PreviousNext
Powered by