Hey all, I am trying to get the metaflow “Triggering events based on other flows” working, however, when I run

FirstFlow

, I get this error:

Unable to publish Argo Event (metaflow.FirstFlow.end): unknown url type: 'None'

Can someone tell me what did I miss with the setup?

a couple of questions - 1. did you deploy argo-events in your infra? 2. if so, did you configure your local deployment using

metaflow configure kubernetes

?

1. Yes 2. No, I used env vars. The ones I used are:

export METAFLOW_KUBERNETES_NAMESPACE=argo-events
export METAFLOW_ARGO_EVENTS_EVENT=metaflow-event
export METAFLOW_ARGO_EVENTS_EVENT_SOURCE=argo-events-webhook
export METAFLOW_ARGO_EVENTS_SERVICE_ACCOUNT=default
export METAFLOW_KUBERNETES_CONTAINER_IMAGE=<some-image>

Did I miss any more configs?

I think you are missing

METAFLOW_ARGO_EVENTS_WEBHOOK_URL

. This should point to the Argo-events webhook service endpoint inside your cluster.

@proud-eye-90172 so I tried that, but it still doesn’t work. I am not sure what I did wrong. So let me tell you what I have and maybe we can pin point the issue. So I have my

EventSource

defined as this:

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: argo-events-webhook
  namespace: {{ .Release.Namespace | quote }}
spec:
  service:
    ports:
      - port: 12000
        targetPort: 12000
  webhook:
    metaflow-event:
      port: "12000"
      endpoint: /metaflow-event
      method: POST

and my configuration as

export METAFLOW_ARGO_EVENTS_WEBHOOK_URL=<http://argo-events-webhook-eventsource-svc.argo-events.svc.cluster.local:12000/metaflow-event>

However, now I see the error as

Unable to publish Argo Event (metaflow.FirstFlow.end): [Errno 104] Connection reset by peer

Do you know what am I doing wrong?

Can you confirm that the Argo-events webhook pod is running in your argo-events namespace?

You mean the

EventSource

pod right? Yeah I have it in my namespaces as

argo-events-webhook-eventsource-srcfd-5dcbc8745-mqksm

If you mean the

argo-events-webhook

then no, because as I understand, it is only a “Argo Events admission webhook”

Yes.. I meant a pod that was named

argo-events-webhook-eventsource*

. Looks like you already have it. ✅ Can you run an nginx pod in your cluster and try to hit the argo-events webhook endpoint?

$ kubectl run nginx --image=nginx

$ kubectl exec -it nginx bash

curl -d '{"message":"this is my first webhook"}' -H "Content-Type: application/json" -X POST <http://argo-events-webhook-eventsource-svc.argo-events.svc.cluster.local:12000/metaflow-event>

@better-printer-5326: Let me know how it goes. If it doesn't get resolved, happy to jump onto a call to debug this live to save some time. We can summarize the finding here for everyone.

Thanks @proud-eye-90172 for the details. Unfortunately, we use Istio and it is not easy to test connection between 2 pods.. But also, maybe that is the problem. Either way, I will let you know what I find out.

Interesting.. I see a K8s service exposing port 12000.

apiVersion: v1
kind: Service
metadata:
  labels:
    controller: eventsource-controller
    eventsource-name: argo-events-webhook
    owner-name: argo-events-webhook
  name: argo-events-webhook-eventsource-svc
  namespace: svcs-forestbee
  ownerReferences:
  - apiVersion: <http://argoproj.io/v1alpha1|argoproj.io/v1alpha1>
    blockOwnerDeletion: true
    controller: true
    kind: EventSource
    name: argo-events-webhook
spec:
  clusterIP: 172.20.69.65
  clusterIPs:
  - 172.20.69.65
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 12000
    protocol: TCP
    targetPort: 12000
  selector:
    controller: eventsource-controller
    eventsource-name: argo-events-webhook
    owner-name: argo-events-webhook
  sessionAffinity: None
  type: ClusterIP

And this seems to be created by the argo-events controller. Do you want to check in your argo-events-controller logs if there are any errors creating this service. Maybe it's service account doesn't have permissions.

@proud-eye-90172 so I finally was able get over that step. The error was because of some custom service mesh stuff that we use internally. However, now I get this error in `secondflow-sensor-xck6n-86f6868999-dgdtq`:

time="2024-02-14T21:51:29.290Z" level=fatal msg="Failed to submit workflow: rpc error: code = PermissionDenied desc = <http://workflowtemplates.argoproj.io|workflowtemplates.argoproj.io> \"secondflow\" is forbidden: User \"system:serviceaccount:argo-events:default\" cannot get resource \"workflowtemplates\" in API group \"<http://argoproj.io|argoproj.io>\" in the namespace \"argo-events\": Azure does not have opinion for this user."

Which is the sensor that was created by

secondflow

. Is there a way to configure the

secondflow

Sensor object to use a ServiceAccount

argo

?