Hey all, I am testing Metaflow events and trying to use

@trigger_on_finish(flows=["FlowB", "FlowC"])

on

FlowD

. However, it doesn’t seem to trigger

FlowD

, does anybody have any idea of where to look to debug this issue?

@better-printer-5326 do you see events for flow completion for B and C in the sensor logs for D?

Yeah, I see this log which I think is relevant:

{
  "caller": "sensor/trigger_handler.go:143",
  "expr": "metaflowFlowBend_mlu7&&metaflowFlowCend_movu",
  "level": "info",
  "logger": "argo-events.sensor",
  "msg": "Evaluating",
  "parameters": {
    "metaflowFlowCend_movu": true
  },
  "sensorName": "flowd",
  "ts": 1709073796.278123
}

Did it actually succeed to evaluate? If it did, then why didn’t it create the flow?

Argo-events can be complicated to debug ... Here's how I would debug this: • Ensure that when

FlowB

or

FlowC

complete, there is an even published. You can see this in the

stderr

of the

end

step of a flow

Argo Event (metaflow.FlowB.end) published.

• This would mean that the event was sent to the

webhook

pod • Look at the logs of the

webhook

pod for any errors. The

webhook

pod should be writing these events to the eventbus/Kafka. • Confirm at that eventbus/Kafka pods are running. Check if there are any errors there. • For every

@trigger...

that you have, there is a sensor object. And that sensor object has a corresponding deployment. So, look for K8s deployment objects with one replica for that sensor. The name of this deployment will typically start with the flow name like ...

flowdflow-sensor-...

. This deployment has 1 replica. • Take a look at the logs of this pod. This should indicate whether this sensor deployment pod was able to read from eventbus/kafka • This sensor deployment should also indicate whether it was able to trigger

FlowD

or reasons why it couldn't. • And above all, most of these components are setup, monitored and managed by two main controllers: argo-events-controller and argo-workflows-controller. Take a look at the logs of these two pods to see why they didn't start the flow.

Hey all, sorry for not looking into this sooner. Here are my observations to what you mentioned:

• Ensure that when

FlowB

or

FlowC

complete, there is an even published. You can see this in the

stderr

of the

end

step of a flow

Copy code

Argo Event (metaflow.FlowB.end) published.

FlowB end

and

FlowC end

events indeed get produced. I see the events in the logs and in the Kafka topic

• Look at the logs of the

webhook

pod for any errors. The

webhook

pod should be writing these events to the eventbus/Kafka.

No errors in

webhook

, also as mentioned above I see them in the Kafka topic.

• For every

@trigger...

that you have, there is a sensor object. And that sensor object has a corresponding deployment. So, look for K8s deployment objects with one replica for that sensor. The name of this deployment will typically start with the flow name like ...

flowdflow-sensor-...

. This deployment has 1 replica.

Yes, I see sensors

flowb

,

flowc

, and

flowd

with 1 replica

• Take a look at the logs of this pod. This should indicate whether this sensor deployment pod was able to read from eventbus/kafka

Yeah it does. I attached logs.

• This sensor deployment should also indicate whether it was able to trigger

FlowD

or reasons why it couldn’t.

It doesn’t seem to even try to process the trigger. In the other sensors I see a log statement as

Successfully processed trigger 'flowb'

. I don’t see it here. All I see is what I attached in the logs. I haven’t seen any errors in Argo Workflows or Argo Events controllers. Not sure what is wrong honestly.

is there a reason to use kafka and not jetstream? we haven't tested against kafka

Yeah it's what the data infra team provide support for

@better-printer-5326 Have you seen any successful triggers that have worked with Kafka and Argo-events? My guess is that the sensor deployment is failing silently when communicating with Kafka. I see someone else had problems running argo-events with Kafka (not quite the same): https://github.com/argoproj/argo-events/issues/2654. Can you check if updating the permissions on the Kafka side helps?

Yeah. So my setup looks like the attached picture. So FlowB and FlowC were triggered correctly. The only problem is FlowD in here, for some reason it does not trigger. @proud-eye-90172 the issue you linked actually happened to me, all we had to do is add

transactionalId

authz, and then it worked fine afterwards.

👍 Done..