Hi team! I am experiencing an unexpected behavior with Metaflow and ArgoWorkflows, where the flows will belong to a production namespace by default, and no matter what I do. As a result, it is not possible to leverage the user namespaces with remote flows. And this is causing troubles with authorization tokens on occasion, that create friction with my devs. Am I missing something? Should I use

--with kubernetes run

instead of

argo-workflows trigger

?

Hi Fabien! Can you further describe the troubles with the auth tokens? The user namespaces are not propagated to the Argo workflows domain like they are when you run a flow from a laptop or workstation with the

run

command. Instead, devs will see something like this when running

argo-workflows create

command:

To analyze results of this production flow add this line in your notebooks:
    namespace("production:mfprj-dwmhwksvlvcfqfni-0-nxte")

These namespaces are isolated for each deployment on Argo workflows and do the same thing as the

user:<mailto:dev@metaflow.org|dev@metaflow.org>

namespace does when you interactively launch flows using

run

.

I see, so it's expected the user namespaces aren't propagated to argo workflows. I was probably confused with the fact that argo workflow templates created by Metaflow are named using the username, although it is not possible to filter using this namespace when using the Metaflow Client API. I encountered two types of surprises with this. 1) not being able to filter user flows. I lead a team of devs who are deploying their own features for testing, and it was confusing not to find their namespaces through Client API. 2) when deploying a series of interdependent flows all at once, no problem. When redeploying just one flow, the namespace seems to change, and then it can break how the flows work together. E.g. like if one flow was working with an older version of another. I am not sure how metaflow tokens are generated, although it looks like all my flows are using the same token if I deploy in a batch (from the same process? sequential deployment automated for all flows in a directory). If I deploy just one flow at a different time (and therefore from a different machine, we're working from cloud VMs) the token would be different, hence the flow may not belong to the same namespace?

Ya, your description all seems correct to me. For 2, is there a way to batch the deployment of interdependent flows? then when one needs to access contents of another, the relevant namespace will be default. If you need to find old namespaces and this won't work, then you can use namespace

namespace(None)

and filter for the most recent run of the old flow in a namespace starting with

production

.

Yes, we're looking into batch deployment by default. For prod flows it's not an issue, we're automating that. For dev flows, it can take several minutes for ~10 flows to get deployed sequentially. This is causing some friction. Do you foresee any blocker/issue of parallelising deployment? (e.g. using

xargs

or

parallel

on the batch of flow files)

I think that would work, but haven't tested exhaustively. Lmk if you hit issues with it and we can debug on a call this week.

First tentative looks to work quite fine 🙂

great!