Hi, im trying to get triggering flows from argo events working but whenever I publish an event I see a webhook pod running and within it's log i see my event, but the flow i created which are supposed to trigger from the event never runs. I have the following setup so far:

"METAFLOW_ARGO_EVENTS_EVENT": "xyzxyz",
    "METAFLOW_ARGO_EVENTS_EVENT_BUS": "default",
    "METAFLOW_ARGO_EVENTS_EVENT_SOURCE": "argo-events-webhook",
    "METAFLOW_ARGO_EVENTS_INTERNAL_WEBHOOK_URL": "webhook-eventsource-test-svc.argo-events.svc.cluster.local",
    "METAFLOW_ARGO_EVENTS_SERVICE_ACCOUNT": "operate-workflow-sa",
    "METAFLOW_ARGO_EVENTS_WEBHOOK_URL": "<https://argo-events.prod>.*******.tech/example"

My flow looks like this:

from metaflow import FlowSpec, step, trigger, pypi
import datetime

@trigger(event='jri-test')
class TestEventFlow(FlowSpec):

    @pypi(python='3.11.3')
    @step
    def start(self):
        print(f"I just got triggered from the jri-test event {datetime.datetime.now()}")
        self.next(self.end)

    @step
    def end(self):
        print("finished...")
        pass

if __name__ == '__main__':
    TestEventFlow()

After deploying the flow with this:

python test-flows/event-triggered-flow.py --environment=conda argo-workflows create

I see a sensor+trigger in the UI, so i guess it works as supposed, however i do think that I have something in my configuration which does not work. argo workflow are under the namespace

argo

and argo-events are under

argo-events

if that has something to do with it. And publish the event using this on my local machine:

python -c "from metaflow.integrations import ArgoEvent; ArgoEvent(name='jri-test').publish();"
# and i also tried this
python -c "from metaflow.integrations import ArgoEvent; ArgoEvent(name='xyzxyz', payload={'name': 'jri-test'}).publish();

I attached screendumps of the webhook pods. And I never see my flow run. I hope someone might be able to help 🙏

There are a few components that work in tandem to make this work. • When you create a flow which has the

@trigger

decorator, it will create a

sensor

object in K8s • For each sensor object, there is a

sensor-deployment

. You can see the sensor-deployment pod. Check it's logs to see if anything is wrong • This sensor-deployment pod subscribes to events that are published on the

eventbus

. This could be NATS, Jetstream, Kafka, etc. I believe the default is jetstream. And you might see a statefulset with pods

eventbus-0

,

eventbus-1

, etc. • When you publish an event, the http request is sent to the

webhook

pod which writes it to the eventbus. The sensor-deployment reads the event from the eventbus and actually creates the argo-workflow. In your case, most likely the sensor-deployment pod will show the reason why the argo-workflow could not be created. Maybe an RBAC issue in creating the workflow object.

We had a similar difficulty getting event triggering to work at first, and what fixed it was using the same value for

METAFLOW_ARGO_EVENTS_EVENT_SOURCE

and

METAFLOW_ARGO_EVENTS_EVENT

. I am sure there is a good reason why these exist as separate parameters but IMO they could use a bit more documentation.

@User thanks for your reply. That certainly helped me to get a bit further with my investegation. I now see that i do not get a sensor-deployment and i suspect it has something to do with our setup because we have argo-events in

argo-events

namespace and we have workflows in a namespace called

argo

. I have this set in my metaflow configuration:

"METAFLOW_KUBERNETES_NAMESPACE": "argo"

and now i tried to modify it to

argo-events

and then i got a deployment for my sensor, but I want the workflow that triggers to run in the

argo

namespace. How do I get that behaviour? I found this PR which seems to address this: https://github.com/Netflix/metaflow/pull/1463 @quiet-afternoon-68940 do you run argo-events in a seperate namespace than where you run your workflows?