Hello I m trying to connect to the metaflow UI without port Outerbounds #ask-metaflow

Hello! I'm trying to connect to the metaflow UI wi...

rhythmic-ghost-56165

02/16/2024, 5:03 PM

Hello! I'm trying to connect to the metaflow UI without port-forwarding on GCP. I have a https hosted instance of the metaflow ui but seeing the following error

Copy code

Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    Metaflow service error:
    Metadata request (/flows/HelloFlow) failed (code 405): 405: Method Not Allowed

The following is my config

Copy code

"METAFLOW_DATASTORE_SYSROOT_GS": "gs://[bucket]/tf-full-stack-sysroot",
    "METAFLOW_DEFAULT_DATASTORE": "gs",
    "METAFLOW_DEFAULT_METADATA": "service",
    "METAFLOW_KUBERNETES_NAMESPACE": "metaflow",
    "METAFLOW_KUBERNETES_SERVICE_ACCOUNT": "ksa-metaflow",
    "METAFLOW_SERVICE_INTERNAL_URL": "<http://metadata-service.metaflow:8080/>",
    "METAFLOW_SERVICE_URL": "https://[host]/api/metadata/"

I am able to get the following from curling the service

Copy code

curl https://[host]/api/metadata/flows
[{"flow_id": "TensorflowFlow", "user_name": null, "ts_epoch": 1708092870596, "tags": null, "system_tags": null}]%

I've seen a few posts like this but no exact fixes mentioned. Can anyone help?

👀 1

hallowed-glass-14538

02/16/2024, 8:09 PM

hey! What is the version of the metadata-service and what is your version of metaflow ?

rhythmic-ghost-56165

02/16/2024, 9:08 PM

Thanks for the reply @little-apartment-49355 Metaflow 2.9.14 i've tracked back to these image versions after seeing it suggested that they worked for this issue in a previous post

Copy code

variable "metaflow_ui_static_service_image" {
  type    = string
  default = "public.ecr.aws/outerbounds/metaflow_ui:v1.0.1"
}


variable "metadata_service_image" {
  type    = string
  default = "public.ecr.aws/outerbounds/metaflow_metadata_service:v2.2.4"
}

variable "metaflow_ui_backend_service_image" {
  type    = string
  default = "public.ecr.aws/outerbounds/metaflow_metadata_service:v2.2.4"
}

rhythmic-ghost-56165

02/17/2024, 9:04 AM

@little-apartment-49355 @square-wire-39606 @straight-shampoo-11124 Have moved to the image versions in the outerbounds example repo. When I look at the backend service I see two requests executing the flow (without port forwarding):

Copy code

INFO:aiohttp.access:10.246.0.6 [17/Feb/2024:08:58:00 +0000] "GET /api/flows/HelloFlow HTTP/1.1" 404 324 "-" "python-requests/2.31.0"
INFO:aiohttp.access:10.246.0.6 [17/Feb/2024:08:58:01 +0000] "POST /api/flows/HelloFlow HTTP/1.1" 405 210 "-" "python-requests/2.31.0"

Presumably the post leads to the following error because only get requests are allowed:

Copy code

Metadata request (/flows/HelloFlow) failed (code 405): 405: Method Not Allowed

Now when I run with port forwarding, I get two GET requests like this:

Copy code

INFO:aiohttp.access:10.246.0.7 [17/Feb/2024:08:59:28 +0000] "GET /api/flows/HelloFlow/runs/3/metadata?step_name=start HTTP/1.1" 200 731 "<https://metaflow-mlops-bidderml-staging-01.bluecaffeine.io/?_group_limit=30&_limit=30&_order=-ts_epoch&status=completed%2Cfailed%2Crunning>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
INFO:aiohttp.access:10.246.0.7 [17/Feb/2024:08:59:28 +0000] "GET /api/flows/HelloFlow/runs/3/metadata?step_name=start HTTP/1.1" 200 731 "<https://metaflow-mlops-bidderml-staging-01.bluecaffeine.io/?_group_limit=30&_limit=30&_order=-ts_epoch&status=completed%2Cfailed%2Crunning>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"

...Any thoughts what the cause might be? Or a solution?

rhythmic-ghost-56165

02/17/2024, 9:12 AM

looks like the requests that are meant to be going to the metadata-service are going to the backend service

Copy code

INFO:aiohttp.access:127.0.0.1 [17/Feb/2024:09:06:07 +0000] "GET /flows/HelloFlow HTTP/1.1" 200 290 "-" "python-requests/2.31.0"
INFO:aiohttp.access:127.0.0.1 [17/Feb/2024:09:06:07 +0000] "POST /flows/HelloFlow/run HTTP/1.1" 200 435 "-" "python-requests/2.31.0"

rhythmic-ghost-56165

02/17/2024, 11:50 AM

Ok I think the cause in this case is because I don't have a second ingress for the metadata-service. Does anyone have any recommendations for this?

hallowed-glass-14538

02/17/2024, 4:46 PM

Hey Bayan, sorry for the delay. If you are using the same endpoint as the “UI service” (api which powers the UI) then you will not have access to the write endpoints and only read endpoints. You will need to expose the Metadata service to the Metaflow. A second ingress is a good start. A quick hack can also be a port forward to you local machine from the metadata service.

rhythmic-ghost-56165

02/18/2024, 8:42 PM

@little-apartment-49355 Thanks, I arrived at the same conclusion, what would you recommend in gcp for this?

2 Views

Open in Slack

Previous Next