Let's say I previously ran Metaflow GCP terraform against the default region (us-west2). If I modify the region/zone in https://github.com/outerbounds/metaflow-tools/blob/master/gcp/terraform/variables.tf to a different one, will it "just work" to install Metaflow infra in the other region/zone? Should I have it use a different GCS bucket as well? I noticed the instructions on terraform workspaces, but I'm not certain if that pertains to having multiple Metaflows in different regions, of if it's only needed when Metaflow is installed multiple times in one region.

1. if you change the TF local variable for region to another region, and reapply, it will try to update your existing cloud resources to conform to the new region. If in-place update is not possible (typically from GCP limitations), it will destroy and recreate resources. 2. if the goal is to have separate deployments, TF workspaces is a good way (1 workspace == 1 deployment). If the thing distinguishing these separate deployments is "region", it would be best to convert the "local" value for region into a "var", and then having a separate tfvars file (listing the var choices), one file per deployment/workspace.

Thanks, @silly-spring-73517! Another question I have is that I noticed that the TF template makes a zonal cluster for Metaflow. Is there an easy tweak to make it regional instead?

1. zonal cluster - yes can just override

location

in your TF code. https://github.com/outerbounds/metaflow-tools/blob/11f01444690f5189f95ccd49e735a4341b697b82/gcp/terraform/infra/kubernetes.tf#L13. Note changing this value likely is a "delete/re-create" operation existing clusters 2. For pointing to existing cluster, the OSS templates does not support that seamlessly. Likely things to change would be here and here. Also trace all usages of this data object.

ok, I'll take a look! Is (2) any easier with the generic kubernetes template (not GCP/GKE), or does it have the same issue?

Which generic kubernetes templates are you referring to?

Yes I think that might be a better fit for "existing kubernetes cluster" use case. We do not have a one touch experience for that (that is tested regularly). Yes will need some a bit more setup. 1) install the helm charts on the cluster, 2) ensure everything comes up. 3) portforward. 4) ensure metaflow config json is setup right (right service URLs etc.). One complication is that GKE uses cloud SQL proxy side car, which is not accounted for in those generic helm charts.

(may be other complications)

thanks. I'll fiddle and see what can be done 🙂

Init is not related

Service accounts are global in GCS. It looks like terraforms gcp provider is confused about this point (it still takes in location to create a service account). Need to look more. If it’s all the same to you fastest way forward is just to start fresh, new workspace and new org prefix (after destroying current one)

Looking closer.... after changing the region, are you operating from the same terraform state? i.e. exact same repo folder? Or was it a new clone?

I was trying again from the same terraform state

Are you using just local tf state? for creating service account in TF, we do not explicitly pass in a location (i was wrong about that). This makes it even more unexpected that changing top level region should trigger TF to want to change anything about service account at all. In terms of nothing to destroy, that is consistent with TF thinking it needs to create the service account again (even though there exists one already in GCP. Where is your tf state? IT should not be empty (nothing to destroy implies empty state).

terraform.tfstate has a bunch of stuff in it. it's on the local filesystem of cloudshell.

And yet all the resources are still present in GCP? E.g. GKE cluster, db, service accounts etc?

you can run

terraform state list

to see what resources ARE known by TF.

(depending on your goal - could be quicker to just start a fresh project). New project is enough (not a brand new gcp org)

Yes, all the resources were there. I'm manually removing everything now because standing up a new project is not an option.

You can consider reverting the region and doing

terraform destroy

(in case this works it will save you time manually removing stuff)

The same tfvars files must be used across

apply

,

destroy

etc (could be the gotcha that got you the '0 things to destroy'

oh haha. I had forgotten! That would have been a good idea 🙂

yeah it's pretty finicky.

updates: • manually deleting everything worked (although a bit painstaking b/c it took a number of iterations to figure out which things depend on the vpc before it can be deleted • moved terraform backend (state) to GCS • used separate terraform workspace from default • changed the region/zone (I'm putting on the back burner using a seprate cluster for k8s jobs and using a regional cluster for metaflow services, so I did it all in one zonal cluster for now) • Changed DB generation • re-ran TF in new workspace • Added new nodepool that has the GPUs I want in region, installed the nvidia driver daemonset • Now I can run a flow with @kubernetes steps that use GPU \o/

That's awesome. BTW what did you mean to paste?

that's weird. I don't know even where that's coming from!