Hi, I deployed my stack on GCP this way

terraform apply -target="module.infra" -var-file=myenv
terraform apply -target="module.services" -var-file=myenv

and everything was deployed perfectly and I can access to the services and run a DAG. But when I try to modify the stack by

terraform plan/apply/destroy -var-file=myenv -target="module.infra"
terraform plan/apply/destroy -var-file=myenv -target="module.services"

I get the following and can't access the services on k8s. I have a

metaflow_gsa_key

service account JSON generated at the infra creation in my

terraform

folder. I tried

gcloud auth application-default login
gcloud container clusters get-credentials mygke --region= myregion

but no luck. Is there anything I'm missing?

╷
│ Error: Get "<http://localhost/api/v1/namespaces/argo>": dial tcp [::1]:80: connect: connection refused
│
│   with module.services.kubernetes_namespace.argo[0],
│   on services/argo.tf line 1, in resource "kubernetes_namespace" "argo":
│    1: resource "kubernetes_namespace" "argo" {
│
╵
╷
│ Error: Get "<http://localhost/api/v1/namespaces/default/services/metadata-service>": dial tcp [::1]:80: connect: connection refused
│
│   with module.services.kubernetes_service.metadata-service,
│   on services/metadata_service.tf line 90, in resource "kubernetes_service" "metadata-service":
│   90: resource "kubernetes_service" "metadata-service" {
│
╵
╷
│ Error: Get "<http://localhost/api/v1/namespaces/default/services/metaflow-ui-backend-service>": dial tcp [::1]:80: connect: connection refused
│
│   with module.services.kubernetes_service.metaflow-ui-backend-service,
│   on services/metaflow_ui_backend_service.tf line 122, in resource "kubernetes_service" "metaflow-ui-backend-service":
│  122: resource "kubernetes_service" "metaflow-ui-backend-service" {
│
╵
╷
│ Error: Get "<http://localhost/apis/apps/v1/namespaces/default/deployments/metaflow-ui-static-service>": dial tcp [::1]:80: connect: connection refused
│
│   with module.services.kubernetes_deployment.metaflow-ui-static-service,
│   on services/metaflow_ui_static_service.tf line 1, in resource "kubernetes_deployment" "metaflow-ui-static-service":
│    1: resource "kubernetes_deployment" "metaflow-ui-static-service" {
│
╵
╷
│ Error: Get "<http://localhost/api/v1/namespaces/default/services/metaflow-ui-static-service>": dial tcp [::1]:80: connect: connection refused
│
│   with module.services.kubernetes_service.metaflow-ui-static-service,
│   on services/metaflow_ui_static_service.tf line 58, in resource "kubernetes_service" "metaflow-ui-static-service":
│   58: resource "kubernetes_service" "metaflow-ui-static-service" {
│
╵
╷
│ Error: Get "<http://localhost/api/v1/namespaces/default/serviceaccounts/ksa-metaflow>": dial tcp [::1]:80: connect: connection refused
│
│   with module.services.kubernetes_service_account.metaflow_service_account,
│   on services/service_account.tf line 2, in resource "kubernetes_service_account" "metaflow_service_account":
│    2: resource "kubernetes_service_account" "metaflow_service_account" {
│
╵
╷
│ Error: Get "<http://localhost/api/v1/namespaces/argo-events>": dial tcp [::1]:80: connect: connection refused
│
│   with module.services.module.argo_events.kubernetes_namespace.argo_events,
│   on common/terraform/argo_events/main.tf line 148, in resource "kubernetes_namespace" "argo_events":
│  148: resource "kubernetes_namespace" "argo_events" {
│
╵

I would guess something is off with

gcloud container clusters get-credentials..

bit. Cluster API endpoint is not supposed to be "localhost" if you're using a cluster running on GKE. And this command is supposed to generate a kubeconfig for you, which includes that hostname

I guess it was my

GOOLE_APPLICATION_CREDENTAILS

env var, somehow it didn't affect while creating the stack but did to modify🤔

I'm not sure how it is in GCP, but in AWS EKS, depending on the nature of the change, they usually allow update k8s cluster in-place without re-creating it. That works fine for me. Its only when you make some changes that make terraform decide that it has to be destroyed and recreated as part of the plan, i get that error (either on apply or destroy)