Hello, We have installed the metaflow service in our GCP Kubernetes cluster (https://github.com/outerbounds/metaflow-tools/tree/master/k8s/helm/metaflow). Then we created a GCP service account (metaflow-metaflow-service) and we gave it all the required permissions for accessing storage objects (Storage Object Admin) and we used that via setting the env variable

METAFLOW_KUBERNETES_SERVICE_ACCOUNT=metaflow-metaflow-service

. Then, we used that env var

METAFLOW_DATASTORE_SYSROOT_GS=<gs://metaflow-storage>

to define our GCP bucket. Now, when we are running the HelloCloudFlow script (python hello.py run) with the

@kubernetes

annotation, a new pod is being spawned in our cluster that uses the above-mentioned service account (we have checked it by describing the pod) but we are getting this error:

Caller does not have storage.objects.get access

. How else can we authorize that spawned job pod to have access to our GS bucket if the service account itself doesn’t work? We cannot find any other information regarding this issue in the metaflow docs. Thanks

@User can you take a look at this?

Hi Nick. There is a distinction between "Google Service Account" (GSA) and "Kubernetes Service Account" (KSA). GSA's are GCP identities that may be granted google cloud storage perms, etc. However in k8s, when a pod runs it runs with a KSA. Additional setup is required in order for the process inside the pod to be able to assume a certain GSA. In GKE, there is something called Workload Identity. It essentially ties together GSA and KSA. Once setup correctly, it says "inject these GSA credentials into any pod running as this KSA". Then processes running inside such pods will get the GCP perms that the GSA has. This is the pattern we use in metaflow-tools terraform templates. The terraform templates prepares the appropriate GSA G / KSA K, and sets

METAFLOW_KUBERNETES_SERVICE_ACOCUNT=K

in the generated metaflow config.json. It sounds like you would like to bring your own GSA G2 / KSA K2. The simplest way would be to examine how existing G / K are setup in the metaflow-tools terraform templates today, and replicate that (with the divergences you need on G2's perms). Other alternatives usually involves you injecting GSA creds into the pod through some other mechanism. E.g.

METAFLOW_KUBERNETES_SECRETS=X

means the k8s secret X will have its keys injected as environment variables of the metaflow task pod.