Hi Team, I'm back. This is what I want to do: Deploy Metaflow & Argo Workflows on GKE with proper scaling and remote access for a limited set of inbound users doing development and deployment I like the Helm chart for metaflow services Outerbounds provides! My thinking so far looks like this: 1. Deploy Argo, ArgoEvents and needed Kubernetes assets into my cluster with helm using Terraform 2. Argo, Argo Events and Kubernetes need to be talking to Metaflow through configurations that are set up in Terraform a. I will need to deploy Argo and Argo events, these will receive webhooks that Metaflow produces, and that's configured....somewhere. b. All of this communicates over a local VPC c. VPC has a load balancer and or proxy pointed outwards to allow my users to authenticate with a google auth and then port-forward using kubectl port-forward {metaflow-service} (?) not clear how/if I need to do this - d. User workflow: i. gcloud auth ii. gcloud container clusters get-credentials {my-deployment} iii. point their local ~/.metaflowconfig/config.json to the deployment setup (I would provide this to my users) iv. Now they should be able to execute on demand via @kubernetes decorator over the remote deployment? e. I assume I need to populate the values here but it isn't super obvious.

cool! It sounded like you got the basic Terraform setup deployed successfully before, right?

Yes, and I like the Helm chart better. It would be cool if I could make that happen

of course! We will follow up tomorrow if it's ok with you 🙂

@victorious-lawyer-58417 if you are watching this thread, I made updates to the original thread. I'm going to start working on a terraform pattern to deploy these ideas into a test cluster. I will post the link up here as each step comes into being

Hi Jared - I may be able also float some ideas here - may be later this PM (PST)

Hi Jared - new to this thread! Want to understand your situation a bit better. Have you tried to use https://github.com/outerbounds/metaflow-tools/tree/master/gcp/terraform out of the box? Based on your requirements description above, I think the main thing missing is a production grade auth solution (e.g. when folks want to check MF UI, may be they don't want to port forward all the time, etc).

@User hi Jackie. Yes I have the terraform plan stood up and it seems to work well. Two items to that: 1. We want to have the option to deploy it in an existing cluster, so I need to be able to do that at the very least. 2. Yes, the auth solution is a barrier. I have a lot of ideas but not a lot of experience doing something like this and would like to minimize the number of false starts and dead ends.

On 1. I also saw that you are interested in using Helm charts of the metaflow services piece. It should all be doable. However some likely obstacles (which are workable) are: A: Metadata service and MF UI backend services need to talk to the database. The standard pattern in GKE is the CloudSQL proxy. The helm charts may need to be modified to support that sidecar pattern. B: As part of A, and possibly to allow storage blob access from the UI backend service, you will have to workload identities. C: Argo / Argo Events stuff - as you have already identified, those are not part of the MF Helm charts. You would install them separately. Expect some amount of debugging there! For all these, https://github.com/outerbounds/metaflow-tools/tree/master/gcp/terraform can still be a good reference (we solve A,B,C in there).

@User perfect, yes. As I said to Ville, I want to do this in two parts A: Get the existing Outerbounds terraform set up deployed to satisfy my ticket - this is priority B: Work on getting a helmified version of all of this, obviating the issue discussed in C above, but I think that I can manage that on my own time.

Sure on auth. There are two parts. First is UI access (interactive). Second is Metadata service API access (so local flows running on laptop can talk to it)

Note internally we have not had a chance to do it - but based on our AWS experience... somethign like this wouldwork for UI access. https://cloud.google.com/iap/docs/enabling-kubernetes-howto https://cloud.google.com/iap/docs/concepts-overview

On the metadata service API access side... something like this is promising.

@sparse-agent-19997 I am in the same situation and found your thread here, I was wondering if you would be willing to share your configurations with me? TIA!