👋 A few months back, I asked a question here about deploying Metaflow to GCP. Naturally, after asking the question, I was almost immediately pulled into other projects and couldn't get back to the project until now. So let me try to re-ask in a better way: I'm trying to deploy Metaflow to GCP using https://github.com/outerbounds/metaflow-tools/tree/master/gcp/terraform as my baseline. This code works and I have a metaflow cluster up and running. The problem is I don't know how to expose this cluster to my developers. My ideal solution would be a load balanced endpoint (e.g. metaflow.example.com) that correctly routes their requests to the UI/API/metadata service/etc that is managed as part of our terraform for deploying metaflow generally. If I'm reading docs correctly, there are three missing parts in the current GCP example deployment: 1. The load balancer (IP, url map, other objects) 2. The k8s ingress controller holding the routes to the actual services 3. The backend service/network endpoint that connects 1 & 2. 1 is easy. There are plenty of examples of creating load balancers in GCP. 2 is harder. I have yet to see a good example of "here are the routes you need to put Metaflow behind a load balancer". Has anyone successfully made one of these (that isn't the not terribly helpful helm chart in the metaflow-tools repo)? 1. Is also a bit confusing to me. It looks like I need a backend service/network endpoint group to point at k8s, but it isn't clear how I actually connect those things. So has anyone else gotten Metaflow deployed in GKE successfully that has any advice for me? I know I can do a lot of this manually, but I'd vastly prefer to have all of this in terraform so we can more consistently reproduce the infrastructure.

below i edited the picture from the AWS doc to show what you want to achieve in GCP/K8S world, not sure if thats helpful:

Right. And the service is for access by actual experiments/etc.

<http://metaflow-service.example.com|metaflow-service.example.com>

will be the programmatic endpoint metaflow-the-python-library talks to programmatically

So I could theoretically have

<http://metaflow.example.com/ui|metaflow.example.com/ui>

and

<http://metaflow.example.com/service|metaflow.example.com/service>

do similar things?

theoretically yes practically this is kind of off the beaten path, good chance you'll be the first person trying to set it up like that :) I'm suggesting two domains option since I've set it up many times and wrote that helm chart with that in mind

IAP does actually support OIDC and token-based auth. I've gotten that working before.

nice! yeah we mostly do this on AWS, and while it may be possible through AWS ALB too (not 100% sure), its would be too convoluted for us so we gave up

I could still do one load balancer with host-based routing (which I have to assume you also do), right?

I think you could, not sure how IAP ingress controller does it. There are two ingresses in K8S land but maybe IAP controller is clever enough to serve both out of one "physical" load balancer? I know AWS ALB ingress controller would create two

IAP attaches to the "backend service", not directly to the load balancer. But also not to the ingress controller, so we should be able to point individual services to whatever ingress controller we need

the only tricky part there is you can see in the helm chart, there is routing so that both backend static assets and UI backend API are served from the same host but its two different K8S services under the hood, routed by HTTP path prefix

Yep. Again, IAP attaches to the backend service, which is (kind of) part of the load balancer, not the the actual service the backend service is reaching out to.

hmm let me see maybe i can find one later today.. the gist of it is here https://github.com/outerbounds/metaflow-tools/blob/72c5ecae49952dfc4020c163780ceff[…]92a/k8s/helm/metaflow/charts/metaflow-ui/templates/ingress.yaml

Yep. Okay cool.