https://outerbounds.com/ logo
Join Slack
Powered by
Have anyone come across this error before using th...
# ask-metaflow
a
Have anyone come across this error before using the metaflow-ui ? When i first load the UI page, the top right circle has a green indicator before turning red (waiting for connection)
👀 1
s
@bulky-afternoon-92433 any ideas?
b
based on the errors, it would seem that the api endpoints are serving html content which shouldnt be happening. most likely a routing config issue between static site and backend routes. @adventurous-gigabyte-81428 what is your hosting setup like? are you running separate backend/frontend containers and something in front of them? or are you using the feature where the backend can also serve the static assets
a
Our environment variable passed into the backend task uses this
Copy code
"MF_METADATA_DB_HOST"              = try("${replace(var.rds_master_instance_endpoint, ":5432", "")}", "")
    "MF_METADATA_DB_NAME"              = "${var.database_name}"
    "MF_METADATA_DB_PORT"              = "5432"
    "MF_METADATA_DB_USER"              = "${var.database_username}"
    "PATH_PREFIX"                      = "/api/"
    "MF_DATASTORE_ROOT"                = "s3://${aws_s3_bucket.this.bucket}/metaflow"
    "METAFLOW_DATASTORE_SYSROOT_S3"    = "s3://${aws_s3_bucket.this.bucket}/metaflow"
    "LOGLEVEL"                         = "INFO"
    "METAFLOW_SERVICE_URL"             = "<http://localhost:8083/api/metadata>"
    "METAFLOW_DEFAULT_DATASTORE"       = "s3"
    "METAFLOW_DEFAULT_METADATA"        = "service"
initially we thought there was a problem with passing in
Copy code
CUSTOM_QUICKLINKS
but after removing that env variable, the problem still remain
images used are
Copy code
metaflow_metadata_service_container_image = "public.ecr.aws/outerbounds/metaflow_metadata_service:v2.4.7-5-g810df2b"
metaflow_ui_static_container_image        = "public.ecr.aws/outerbounds/metaflow_ui:v1.3.9-1-g1013015"
Changing the to older images still exhibit the same message
b
@adventurous-gigabyte-81428 is this a new deployment? Or one that was working previously?
a
working previously
b
Did you upgrade?
My first thought would be to look at the DevTools/Network tab in the browser. Look at the requests that are failing and confirm that the URL's domains are the ones that you would expect. Then double-check the network connectivity to those endpoints
a
I'm not familiar with using the tools mentioned. but here's the screenshot of what i think you're referring to
b
The top section is what you want to look at. Clicking on one of those 
ws
lines should give you information about the URL that the websocket is connected to
It would also be worth looking at the non-ws requests
a
Screenshot 2024-02-08 at 09.28.49.png
Screenshot 2024-02-08 at 09.30.07.png
b
In the 
Headers
tab you will be able to see a request URL. Double check that that URL is the URL of your server
a
yes the request URL is of our server
b
How about for the requests that aren't 
ws
?
a
The highlighted blue box all have request URL root from our server via https The highlighted red box have request coming from chrome extension
b
It looks like the connectivity is ok (for non-websockets). The error in the screenshot you shared early would have come from the 
/runs
request in your last screenshot. Click on the 
/runs
line and look at the 
Preview
tab
a
Screenshot 2024-02-08 at 09.46.34.png
b
That tells me the 
/runs
request is not hitting the backend server. It looks like it is hitting a default index file. You can confirm this by right-clicking on the request, copying as curl, and then running that command from your terminal. It seems likely to me that the webserver is not setup correctly
a
Copy code
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Build and manage real-life data science projects with ease."/><link rel="apple-touch-icon" href="/logo192.png"/><link rel="manifest" href="/manifest.json"/><title>Metaflow UI</title><script defer="defer" src="/static/js/main.2152150e.js"></script><link href="/static/css/main.8b8fc6d5.css" rel="stylesheet"><script>window.METAFLOW_SERVICE="/api";window.MF_DEFAULT_TIME_FILTER_DAYS="";</script></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>%
Thats the result running the curl pasted on terminal
when you mentioned the webserver. Are you referring to the ui-static or ui-backend or metadata-service ?
b
did your 
/runs
request start with 
/api
?
a
yes it does /api/runs
b
Just to confirm - the domain isn't localhost?
a
no it isn't localhost. it's our server url
route 53 + ALB + targetgroup that links to the AWS ECS task
b
The web application is being served correctly, so your webserver is working, but the API requests are returning the index page. There is something borken in the routing of paths. Unfortunately I have no more ideas on how to debug that
a
Here's an update, we think we may have resolve it via introducing the routing order in the AWS ALB listener. Previously there is no priority set to the listener rule. After setting this up like https://github.com/outerbounds/terraform-aws-metaflow/blob/master/modules/ui/ec2.tf#L132 and https://github.com/outerbounds/terraform-aws-metaflow/blob/master/modules/ui/ec2.tf#L126 we are able to connect and load metaflow-ui without error.
👍🏽 1
11 Views
Open in Slack
PreviousNext
Powered by