Hey, we want to evaluate Metaflow on GCP and I am checking out deployment options. I see there is a full example provided which includes the deployment of the infrastructure as well. I noticed in metaflow-tools there is also a Helm chart, and I was curious if someone has tried deploying Metaflow to GKE instead? Given there are some hardcoded variables in the template file

- name: METAFLOW_DEFAULT_DATASTORE
              value: "s3"

there should be some care given. Though it seems this could be overwritten by

metaflow-ui.metadatadbEnvVars

. Another question I had -- we don't use Argo but have Airflow with KubernetesExecutor deployed already. As far as I understand the full GCP example, this is not really a problem, since the deployment itself doesn't rely on any Argo settings. However, it seems like config.json is read from the local machine itself (where the flow is ran from). Is there an example showing what settings need to be used to get it working with Airflow? Or is no config needed? Based on the docs I understand a valid CONN_ID env variable suffices. However, we'd still need to come up with an approach of automatically pushing generated DAGs to GCS bucket. So I am again curious if someone has some experience with this? Thanks!

Hey @early-accountant-74956, let me try to answer your questions:

I was curious if someone has tried deploying Metaflow [helm chart] to GKE instead?

• I am not sure if anyone on our team has tried to deploy the helm chart to GKE. However, you should be able to give that the chart is configurable. Please do let us know if you encounter any issues and we push out fixes.

Is there an example showing what settings need to be used to get it working with Airflow? Or is no config needed?

• @hallowed-glass-14538 is the original author of this doc, so maybe he can shade some light here, but according to the doc:

Metaflow can work with any Airflow executor

.

hey Matija, are you deploying the metaflow helm chart after deploying the GCP terraform template ? can you share the template you are using ? The GCP terraform template has a deployment of services in the kubernetes cluster and those will configure the services to GCP. WRT deploying on KubernetesExecutor: yes, metaflow's Airflow integration will work with a k8s executor. But that said there are certain things to note on how airflow behaves with the kubernetes executor:

The KubernetesExecutor creates a pod for each airflow task. Lets call this Pod A. Since our Metaflow steps get compiled to KubernetesPodOperator, Pod A will deploy another pod which executes the actual Metaflow Task.

Since Pod A is executing a Airflow task, Pod A requires all the dags to be present in the pod. So you will have to determine a way to pass dag files to Pod A. Airflow recommends many ways to pass the dag files to the pod;