Hi Metaflow, @wide-battery-59962 and I have been testing out Metaflow on EKS + Airflow, and one of the issues we’re running into is that the airflow dag fails because it is unable to find credentials within the pod.

[2022-12-19, 20:35:06 UTC] {{pod_manager.py:197}} INFO - Downloading code package...
[2022-12-19, 20:35:12 UTC] {{pod_manager.py:197}} INFO - fatal error: Unable to locate credentials
[2022-12-19, 20:35:22 UTC] {{pod_manager.py:197}} INFO - Failed to download code package from s3://*** after 6 tries. Exiting...
[2022-12-19, 20:35:23 UTC] {{pod_manager.py:337}} INFO - Running command... cat /airflow/xcom/return.json

I should note that we configured an irsa_role on the cluster with a kubernetes service account and used that to set

METAFLOW_KUBERNETES_SERVICE_ACCOUNT

. We are also using MWAA from AWS, and setting

incluster = False

after dag.py has been created.

Unable to locate credentials

seems to be the

NoCredentialsError

message within the awscli package. Is there a metaflow argument to avoid trying to instantiate credentials and just make the AWS API calls which will pick up the IRSA role in the CNI?

Hey can you check if the

CONFIG

object in the dag.py file has the same service account defined as you had set in METAFLOW_KUBERNETES_SERVICE_ACCOUNT ? Even grepping service_account can give some insight.

Yeah it is getting into that

Does the pod have any aws related env variables set via irsa ?

The SA is annotated as expected similar to other irsa workloads