https://outerbounds.com/ logo
Join Slack
Powered by
Argo Workflows / Argo Events feature request threa...
# dev-metaflow
a
Argo Workflows / Argo Events feature request thread - my team and I ended up having to move Argo Events into the same namespace as Argo Workflows and I have some follow-up questions about how to catalog future work
✅ 1
Firstly, it seems to be implied by this comment that this is expected, but I'm having a hard time convincing myself that separating Argo Events and Argo Workflows constitutes a "multi-namespace Workflow" in the sense meant by the Argo Workflows issue.
Secondly, even though the comment says support is coming in 
v3.4
, it looks like it's still outstanding (even though that version is out), so, if the two are the same, then that means it's still blocked.
Now the questions. Is the use case that I'm describing (Argo Events in one k8s namespace, Argo Workflows in another) blocked by the linked Argo Workflows ticket? If not, is there value in ticketing support for this use case in Metaflow, at this time?
Per this comment, @square-wire-39606 ^^^
s
@abundant-football-23028 the comment references running an argo workflow across multiple namespaces and clusters (and has no connection to argo events). when the argo workflows implementation was shipped, there was indication that such support for argo workflows execution will land in argo-workflows-v3.4. unfortunately, that never happened. we have since implemented a work around for that functionality within 
outerbounds
. re: 
argo-events
in one k8s namespace - are you planning on running the 
argo-webhook
, 
event-bus
and 
sensors
- all in a single but different namespace to the one that deploys the flows?
a
The 
argo-webhook
and 
event-bus
were deployed by the parameterized Helm Chart into the namespace specific to Argo Events, and then adding the 
@trigger
decorator caused the associated 
sensor
to be deployed in the Argo Workflows namespace. With that configuration, the UI showed the 
sensor
as not wired up correctly (compared to the example deployed with the Helm Chart).
Obviously, challenging the assumption inherent in having a configuration called (the one, true) 
KUBERNETES_NAMESPACE
is a larger effort, but I'm curious if there is value in being able to pass in a namespace that is specific to Argo Events for the few integration points (
@trigger
being one of them).
If so, how would it be best to write that up in the GitHub Issues.
s
if you patch 
KUBERNETES_NAMESPACE
at the right places in this file to point to your alternate namespace - are you able to get everything working?
a
I don't think so, because the trigger_workflow_template assumes that the Sensor WorkflowTemplate and the Metaflow WorkflowTemplate are in the same namespace. Otherwise, I would expect this line to need to include some more qualification, to support referencing across namespaces. This is also what made me think it was limited by support in Argo Workflow for multiple namespaces; i.e.: even if there were a namespace qualifier on that 
workflowTemplateRef
, it wouldn't work because Argo Workflows doesn't support that, yet.
s
trigger_workflow_template
is used to trigger the workflow outside of Argo Events by directly invoking the kubernetes API to launch an already deployed workflow template
internally, we actually do run sensors, event buses and webhooks not only in a different namespace but in a different cluster all together without any changes to open source Metaflow, but that relies on some of our internal tech that's not available in open source.
a
Ok, fair enough.
Thanks for the detailed explanation. We're ok with the work-around of consolidating namespaces, just curious if there was something I could document as a ticket.
7 Views
Open in Slack
PreviousNext
Powered by