Hello all, I am trying to deploy my flows on airflow using GKE. My google_cloud_default seems to be working fine as i am able to start clusters and pods using GKECreateClusterOperator and GKEStartPodOperator but when exporting my flow to airflow using the following command: python parameter_flow.py --with retry airflow create parameter_dag.py (described here: https://docs.metaflow.org/production/scheduling-metaflow-flows/scheduling-with-airflow) the output DAG uses KubernetesPodOperator which is unable to find my gcp default config (see traceback below). Is there any way i can force the airflow create command to produce a dag using GKEStartPodOperator ? Thank you very much for your help.

Traceback (most recent call last):

File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 536, in execute_sync

self.pod_request_obj = self.build_pod_request_obj(context)

File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 851, in build_pod_request_obj

"airflow_kpo_in_cluster": str(self.hook.is_in_cluster),

File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 259, in is_in_cluster

self.api_client  # so we can determine if we are in_cluster or not

File "/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36, in __get__

value = obj.__dict__[self.func.__name__] = self.func(obj)

File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 267, in api_client

return self.get_conn()

File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 209, in get_conn

config.load_incluster_config()

File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 121, in load_incluster_config

try_refresh_token=try_refresh_token).load_and_set(client_configuration)

File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 54, in load_and_set

self._load_config()

File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 62, in _load_config

raise ConfigException("Service host/port is not set.")

kubernetes.config.config_exception.ConfigException: Service host/port is not set.

Hey Hugo, it seems the issue is because the KPO is unable to find the kubeconfig file. This are a few ways around this; Metaflow exposes some config variables that allow specifying either an airflow connection or specifying path to kubeconfig file I think this can help overcome the issue you manybr facing.

Hey Valay, thanks for your reply ! i've added the path to my kube config file in the kubernetes_default connection but now i am experiencing another error: "pods is forbidden: User \"system:anonymous\" cannot create resource \"pods\" in API group \"\" in the namespace \"default\""

If you know your service account then you can specify it in your Metaflow config file like

METAFLOW_KUBERNETES_SERVICE_ACCOUNT

Hey Valay, sorry for the late reply. Turns out i did not have any secret token inside my kubeconfig file so my KPO was unable to push any tasks to GKE. Thanks a lot for your help !

Hey @happy-monkey-82439! Apologies for digging out this old thread, but I am facing some similar problems and wanted to ask - are you launching your Metaflow tasks/workloads on the same GKE cluster that runs the AirFlow scheduler?