Is it possible to configure Metaflow to use Argo Workflows when deployed in single-namespace mode? I have Argo Workflows deployed with access to one namespace (uses Roles/RoleBindings instead of ClusterRoles/ClusterRoleBindings) When I run a FlowSpec, I get the following:

<http://workflowtemplates.argoproj.io|workflowtemplates.argoproj.io> "minimalmlflow" is forbidden: User "system:serviceaccount:ml:metaflow" cannot get resource "workflowtemplates" in API group "<http://argoproj.io|argoproj.io>" in the namespace "ml": RBAC: [<http://clusterrole.rbac.authorization.k8s.io|clusterrole.rbac.authorization.k8s.io> "argo-workflows-workflow-controller-cluster-template" not found, <http://clusterrole.rbac.authorization.k8s.io|clusterrole.rbac.authorization.k8s.io> "None-workflow-controller-cluster-template" not found]

Is there a configuration somewhere where I can tell Metaflow to use a role instead of clusterrole?

hmm Metaflow itself doesn't really care about roles, it just uses whatever kubernetes credentials there are. I think Argo is mostly the same way? I also wonder where that bit comes from:

RBAC: [<http://clusterrole.rbac.authorization.k8s.io|clusterrole.rbac.authorization.k8s.io> "argo-workflows-workflow-controller-cluster-template" not found, <http://clusterrole.rbac.authorization.k8s.io|clusterrole.rbac.authorization.k8s.io> "None-workflow-controller-cluster-template" not found]

i don't think its Metaflow or standard k8s apiserver error message

I had argo deployed cluster wide and then adjusted it to be namespace scoped, I think there are some lingering clusterolebindings and clusterroles I need to delete. Maybe those take precedent if present of Roles and RoleBindings?

yeah i'd look at the rolebindings/clusterrolebindings for ml:metaflow , might be something off there. And check if any of those provide permissions on workflowtemplates like the message says

Yep that was it!