https://outerbounds.com/ logo
Join Slack
Powered by
Hi guys, thank you for all the work you do. I just...
# ask-metaflow
a
Hi guys, thank you for all the work you do. I just have a simple question regarding the update process for Metaflow. I currently use version 2.4.3 and have gone through the patch notes since then, but wanted to make sure if there are any other important information before upgrading to latest (2.12.35+)? Thanks
a
Hi Filip, you can safely upgrade from 2.4.3 to any version. We guarantee strict backwards compatibility (modulo bugs) and often go to great lengths to ensure that.
🙌 1
a
Hi @square-wire-39606, thanks but I unfortunately encountered a problem. Namely, I have a OIDC provider side container running on port 3000 already so I had to change PORT variable for ui-service. After I did that, I encountered this error. The UI service constantly tried to re-establish connection to websocket but that is clearly not configured well on my side. What should these calls point to(meta-data service or ui-service itself)? Also, could you please tell me the difference between METAFLOW_SERVICE and METAFLOW_SERVICE_URL?
s
@bulky-afternoon-92433 would you happen to know?
b
I'll get back to you on the env vars once I'm back at a computer. the websocket requests should be going to the ui-service
a
thanks, I am not sure if there is a problem related to kubectl port-forward
whats also very strange to me is that ui-service sends websocket requests to the ui-service itself? Shouldnt it go to some backend, or in this case metadata service?
t
some follow up questions on your setup so I can debug this • where are you seeing 
METAFLOW_SERVICE
as an env var? Through a quick lookup, 
METAFLOW_SERVICE_URL
should be the only one that's actively used • for deploying the services, are you using some of the existing IaC templates or doing everything manually? • which versions of the services / ui are you trying to deploy? ◦ is this on a remote server, or on your local machine? I'm trying to understand why the websocket is connecting to localhost
a
I used local setup with minikube, but have moved onto our test cluster, so no localhost will be at play anymore. Right now the ui-service pod is constantly killed because livenessProbe tried the PORT, in my case 3333, since I already use 3000.
t
ahh, now it makes sense. I was focused on the metadata-service repo, hence didn't see the env var. the port should just be where nginx in the container is listening on. which route is the livenessprobe trying to check?
a
Copy code
Warning  Unhealthy  3m14s (x12 over 4m23s)  kubelet            Readiness probe failed: Get "<http://IP:3333/ping>": dial tcp 10.2.0.204:3333: connect: connection refused
🤔 1
but when I set containerPort to 8083 it works
and for some strange reason, I get 137 exit code in Status of previous pod, but OOM is impossible, I dedicated 5Gi.
t
as for what the env var does, the intended usage for the container is to run behind a reverse proxy, where the same host serves both the frontend&backend requests. the METADATA_SERVICE is the prefix where requests supposedly get routed to the backend.
a
whats so confusing to me is the fact that the log says:
Copy code
INFO:root:Metadata service available at <http://0.0.0.0:8083/metadata>
Is that info that ui-service container "found" metadata-service?
t
ah, nope. I understand its quite confusing messaging, but heres whats happening; the ui-backend also runs a metadata service of its own, as a read-only flavour. This was needed because under the hood, the ui-backend also uses the metaflow client to process some flow data, and we didn't want to burden the actual metadata service with these requests
as to why the frontend might be working with traffic to 8083, that could be caused by the ui-backend also having the ability to serve the site assets by itself (no frontend container required)
a
Oh, I see. Can this be overriden in the deployment or I have to rebuild the image with it set to 0?
Okay, I see, theres ENV for that. Let me try
b
there's an env for that but the download_ui.sh is run as part of the image build, so you might need to rebuild if you want to disable that. Trying to see what kind of logic was in place for the static asset serving
a
the core issue I have is the fact that we some OIDC sidecar that must run on port 3000. I am certain metaflow-ui would run smoothly otherwise, but by default nginx tries to bind to 3000 which is normally not possible. As soon as I change the port via PORT=3333, and change the containerPort, the liveness and readiness probe fails. I also noted that our previous working state which cloned the repo and built it, had passed the UI_ENABLED argument as well
And yeah, wow, as soon as I changed containerPort to 8083, even though UI_ENABLED=0 and PORT=3333, the pod works but my OIDC sidecar reports an error:
Copy code
[2024/12/19 15:20:16] [error_page.go:91] Error proxying to upstream server: dial tcp 127.0.0.1:3333: connect: connection refused
Sorry for spamming, but with nothing else changed, but the UI_ENABLED=0, I now get 404?
t
from 8083? that would be expected
a
Oh god, I just tried to deploy it without UI_ENABLED=0, nothing else changed and it worked!
Suddenly, there is no conflict with nginx? So strange.
t
hmm. I'll try looking into the frontend container nginx setup later today, a bit hazy on its internals. there might be an issue with the version of nginx thats used
but just to confirm, your deployment is now working?
a
just to confirm, I saw nginx serves static react files and I assume it uses port 3000 for it, but also there is UI metadata service on port 8083? I am a little confused as to how can a container expose two ports? Why does 8083 also serve contents is the right question.
It works now, yes.
b
As far as I can remember, the ability for the ui-backend to serve the static files was introduced to ease the deployment for certain setups that didn't want to introduce loadbalancers/static site hosting in addition to the backend. In these cases deploying just the ui-backend was meant to be a package-deal, with everything included in one process for serving the mfgui.
so there are at least two ways to host the frontend • let the ui-service take care of serving static assets as well • host it separately with the provided docker image from metaflow-ui, and configure routing correctly between ui-static container and ui-service • (third option) host the static files through a separate web server. this still requires configuring routing correctly options 2 and 3 require that same-origin /api routes to the ui-backend correctly.
the only major downside to serving the assets through the backend is that it pins the assets to a specific version, so updating the ui version happens in lockstep with the backend.
a
Ah, so metaflow-ui is actually a image that has both frontend and backend, but metaflow_ui_backend is only "backend" part without frontend? I am looking from here: https://gallery.ecr.aws/outerbounds/
I think I got it, metaflow-ui from this repo is what you are talking about serving separate, but I just didnt find the image published anywhere. I think I understood you.
b
I'll have to check the contents of the images, but going off by version numbers I would assume that https://gallery.ecr.aws/outerbounds/metaflow_ui is published from https://github.com/Netflix/metaflow-ui, so it only contains nginx and the static assets for the frontend whereas any backend image released from https://github.com/Netflix/metaflow-service will have the ability to serve the static assets as well.
4 Views
Open in Slack
PreviousNext
Powered by