Hey folks, is this comment from here still valid?

# Currently, an Argo Workflow can only execute entirely within a single
        # Kubernetes namespace. Multi-cluster / Multi-namespace execution is on the
        # deck for v3.4 release for Argo Workflows; beyond which point, we will be
        # able to support them natively.

Argo Workflow 3.4 has been out for some time now

Depending on your install, Argo Workflows could manage workflows across multiple namespaces in 3.3 too. We are using Argo & Metaflow in this way (on Argo 3.4.4)

Thank you Thomas! We're currently iterating on our setup, standing up a second version to allow for supporting segregated use-cases (e.g. models that touch restricted access data vs models that touch unrestricted access data), since our first attempt did not allow for that. I was told multiple namespaces could be a way to tackle that problem (vs standing multiple segregated instances of metaflow, which is undesirable). Did you follow any publicly available guideline/best practice to get it work for you?

Most of it is on the Argo/Kubernetes side, to be honest. So in our company, each team has a namespace and then we use the service account of the namespace to manage what each team can access. The one thing we haven't done is set up the cluster so that people can only run commands on a given namespace, but again I think there is Kubernetes functionality for that.

Ah, that makes sense I think. I'll report this back to my team (I'm definitely not an expert on kubernetes/argo) and see if this makes sense to them 🙂 Thank you for taking time to answer this question

no problem 🙂

Like Thomas said, its pretty easy to switch namespaces via METAFLOW_KUBERNETES_NAMESPACE. The only thing I'd add is that if you're dealing with data constraints, it sometimes may be desirable to still have segregated metaflow instances. With namespaces only, it can definitely help segregate tasks from Kubernetes RBAC perspective, but then often times that leads to further questions. For example, what to do with data access, like task artifacts and metadata? Ideally you'd want them segregated too, since those may contain same sensitive info, or some derivative of it, which may or may not be ok to share outside of the restricted env. You'd have to be very careful. For these reasons we see a number users start with separate instances and then those instances interact via events and/or some shared data location in S3, that way you have more explicit control on what gets actually shared across those boundaries

@mammoth-rainbow-82717 sorry to bother you directly, just to make sure: do you have a single metaflow instance in some namespace and then have devs able to trigger argo workloads in multiple namespaces by changing that env var?

The metaflow instance (by which I assume you mean metadata-service?) is deployed in ECS. Argo Workflows is deployed in a separate namesapce and then the teams have all of their own namespaces.

Thomas, I'm in the exact same situation and with very similar constraints. We're also trying to stand up the thing in the exact same way you described. Thanks a lot for taking time, once again, to respond to this. Knowing that this is possible and in use at other companies makes me more confident this is the initial path we should take. I'd prefer as well having more isolated instances (yes, metaflow-service(s)) but that also comes with its own set of problems that we're not ready to tackle yet.

I'm curious what those potential problems are in your case? Is it mostly operational issues like Thomas said, i.e. managing multiple instances, creating new ones, database migrations, or something else too?

It's mostly operational issues and making sure that the end experience to the data scientists is consistent across all the instances

@mammoth-rainbow-82717 I'm curious, what ensures that a developer who creates a metaflow workload doesn't set

METAFLOW_KUBERNETES_NAMESPACE

to a namespace he/she shouldn't have access to?

Actually it is the other way around, Argo makes the workflow and then as part of that is registers the Metaflow run. Metaflow service has no concept of Argo, I believe.

@mammoth-rainbow-82717 👋 I saw a few of your posts around ArgoCD/GitOps and was wondering if you could share how you get the flows to your prod environment? I am envisioning the same thing as you have described and currently I am wondering what the best way to get a manifest generated in my gitops repo when a workflow is updated/promoted and would appreciate your insight 🙂

Hey Martin, Sure, no problem. What sort of information do you want to know?

Just that CI part I guess, how you generate the argo workflow manifest to then commit it to your gitops repo for argocd to pick up

I gave a talk at the Metaflow lunch hour series on this topic late last year which covers a lot of the details. Did you manage to see it? @proud-eye-90172 Do you record these session? I can't remember if it was recorded. If there is a recording, I think it would provide a lot of the details you are after @hundreds-midnight-75494

Sadly, we only started using Metaflow late in the year, and I got involved since December only 🙂 happy to look at it, but a high level paragraph explaining the approach would be a great start if I could bother!

Metaflow office-hours sessions are always recorded. The recording of the one that Thomas presented in can be found here:

Awesome! Thanks @proud-eye-90172! @hundreds-midnight-75494 I think this will provide you with a lot of the details you need. 🙂

awesome, thank you!!