Hello all, I am trying to deploy my flows on airf...
# ask-metaflow
h
Hello all, I am trying to deploy my flows on airflow using GKE. My google_cloud_default seems to be working fine as i am able to start clusters and pods using GKECreateClusterOperator and GKEStartPodOperator but when exporting my flow to airflow using the following command: python parameter_flow.py --with retry airflow create parameter_dag.py (described here: https://docs.metaflow.org/production/scheduling-metaflow-flows/scheduling-with-airflow) the output DAG uses KubernetesPodOperator which is unable to find my gcp default config (see traceback below). Is there any way i can force the airflow create command to produce a dag using GKEStartPodOperator ? Thank you very much for your help.
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 536, in execute_sync
self.pod_request_obj = self.build_pod_request_obj(context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 851, in build_pod_request_obj
"airflow_kpo_in_cluster": str(self.hook.is_in_cluster),
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 259, in is_in_cluster
self.api_client  # so we can determine if we are in_cluster or not
File "/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36, in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 267, in api_client
return self.get_conn()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 209, in get_conn
config.load_incluster_config()
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 121, in load_incluster_config
try_refresh_token=try_refresh_token).load_and_set(client_configuration)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 54, in load_and_set
self._load_config()
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 62, in _load_config
raise ConfigException("Service host/port is not set.")
kubernetes.config.config_exception.ConfigException: Service host/port is not set.
1