Hi Team,
I'm back.
This is what I want to do:
Deploy Metaflow & Argo Workflows on GKE with proper scaling and remote access for a limited set of inbound users doing development and deployment
I like the Helm chart for metaflow services Outerbounds provides!
My thinking so far looks like this:
1. Deploy Argo, ArgoEvents and needed Kubernetes assets into my cluster with helm using Terraform
2. Argo, Argo Events and Kubernetes need to be talking to Metaflow through configurations that are set up in Terraform
a. I will need to deploy Argo and Argo events, these will receive webhooks that Metaflow produces, and that's configured....somewhere.
b. All of this communicates over a local VPC
c. VPC has a load balancer and or proxy pointed outwards to allow my users to authenticate with a google auth and then port-forward using kubectl port-forward {metaflow-service} (?) not clear how/if I need to do this -
d. User workflow:
i. gcloud auth
ii. gcloud container clusters get-credentials {my-deployment}
iii. point their local ~/.metaflowconfig/config.json to the deployment setup (I would provide this to my users)
iv. Now they should be able to execute on demand via @kubernetes decorator over the remote deployment?
e. I assume I need to populate the values
here but it isn't super obvious.