Still also with argo i get that an error that it takes the wrong service account in my kubernetes cluster for the pod that executes the python script: Deploying hellocloudflow to Argo Workflows... Argo Client error: workflowtemplates.argoproj.io "hellocloudflow" is forbidden: User "systemserviceaccountcoder:default" cannot get resource "workflowtemplates" in API group "argoproj.io" in the namespace "metaflow" I understand that the jobs can also be created in another namespcae (by argo or kubenetes), when the pyhton (from another namespace or external) is using http to request that. this is the config:

{
  "METAFLOW_ARGO_EVENTS_EVENT": "argo",
  "METAFLOW_ARGO_EVENTS_EVENT_BUS": "default",
  "METAFLOW_ARGO_EVENTS_SERVICE_ACCOUNT": "argo",
  "METAFLOW_DATASTORE_SYSROOT_S3": "<s3://metaflow-test/metaflow>",
  "METAFLOW_DATATOOLS_S3ROOT": "<s3://metaflow-test/data>",
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_KUBERNETES_NAMESPACE": "metaflow",
  "METAFLOW_KUBERNETES_SECRETS": "minio-secret",
  "METAFLOW_KUBERNETES_SERVICE_ACCOUNT": "argo",
  "METAFLOW_S3_ENDPOINT_URL": "<http://minio.minio.svc.cluster.local:9000>",
  "METAFLOW_SERVICE_URL": "<http://metaflow-metaflow-service.metaflow.svc.cluster.local:8080>",
  "METAFLOW_KUBERNETES_WORKFLOW_ENGINE": "argo"
}

I am not sure why, but it appears that your service account

system:serviceaccount:coder:default

is not being set as intended when you run your argo workflow.

it seems i always need a kube-config file (that has the permission to create in argo)? I thought that just over the http metaflow-service url i can schedule my python script from external.

hi tobi - the jobs are submitted not through the metaflow service but by utilizing the credentials available to the user on their workstation. the metaflow service is responsible for tracking the metadata emitted by your workloads and powers the ui - that allows the entire set up to scale out really nicely across 1000s of users and millions of tasks.

Hi Savin, is there a document that describes that? It seems to me that every user needs a kubeconf with permission to create jobs. Dont know if not with oidc I should make sure that external requests are executed securely.

Hey Tobi, each user would indeed need some sort of Kubernetes identity to access the cluster. Metaflow philosophy is to leave this to admin as much as possible so they can use standard tooling and rely on a standard Kubernetes RBAC model, instead of rolling our own authz model and delegating auth decisions to a server, which may or may not match your environment. There are pros and cons to this. But since Kubernetes 1.30 you have a lot of flexibility to set up K8S auth via OIDC directly using structured authentication configs. I'd expect Flyte to make use of this eventually as well. Before 1.30 that it was pretty inflexible and you had only one OIDC IdP per cluster, and for some cloud providers they didn't expose this at all, so I get why they did it this way.