hey folks, looking for information around tying in Argo Events as a trigger for Flows; specifically, I’m looking for what the

kind: EventSource

and

kind: Sensor

should look like. Are there any docs on this?

Hey @careful-grass-53234.. 👋 ... You don't need to explicitly create Argo Events objects such as EventSource or Sensor. You can use the abstractions provided by Metaflow such the

@trigger

decorator.. More info here: https://docs.metaflow.org/production/event-triggering/external-events

I think my process for a lot of the built in abstractions breaks down w/ local development. I do a few things: 1. create minikube 2. deploy metaflow to minikube 3. deploy localstack(s3) to minikube 4. port forward s3 service to localhost 5. port forward metaflow-service to localhost (i use 9080 instead of 8080 bc I have other services forwarding 8080) my

.metaflowconfig/config_local.json

looks like the below:

{
  "METAFLOW_S3_ENDPOINT_URL": "<http://localhost:4566>",
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "METAFLOW_DATASTORE_SYSROOT_S3": "<s3://metaflow-local>",
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_SERVICE_INTERNAL_URL": "<http://ml-ops-metaflow-service.metaflow:9080>",
  "METAFLOW_SERVICE_URL": "<http://localhost:9080>",
  "METAFLOW_KUBERNETES_NAMESPACE": "argo-workflows",
  "METAFLOW_KUBERNETES_SECRETS": "s3-metaflow",
  "METAFLOW_ARGO_EVENTS_EVENT_BUS": "default",
  "METAFLOW_ARGO_EVENTS_EVENT": "metaflow",
  "METAFLOW_ARGO_EVENTS_EVENT_SOURCE": "mf-webhook",
  "METAFLOW_ARGO_EVENTS_WEBHOOK_URL": "<http://mf-webhook-eventsource-svc.argo-events:12000/metaflow>",
  "METAFLOW_ARGO_EVENTS_SERVICE_ACCOUNT": "operate-workflow-sa"
}

Some of the things that are not working for me: 1. the

METAFLOW_S3_ENDPOINT_URL

is being used within the Argo WorkflowTemplate (there likely needs to be an override in the plugin to accept

METAFLOW_ARGO_WORKFLOWS_S3_ENDPOINT_URL

; i may try to work up a PR to support this. a. to get around it, i do something like the below to replace the value in the WorkflowTemplate

METAFLOW_PROFILE=local python3.11 secondflow.py --with retry argo-workflows create --only-json | yq -P | sed s#<http://localhost:4566#http://localstack.metaflow:4566#g> | kubectl apply -f -

2. I am being impacted by https://github.com/Netflix/metaflow/pull/1463, so the sensor gets created in the

argo workflows

namespace, but should be created in the

argo-events

namespace (ie where the default event bus lives) so there are a few things im running into. the

--only-json

does not export the sensor or eventsource YAML, so I am having to piece it all together after the #2 issue above

I see .. I guess

argo-events

is already installed in this case right?

yup! I install argo-events into the argo-events namespace before metaflow is ever created and before flows are triggered

Do you also have an EventSource object created? It should be an eventsource of type

webhook

at port 12000.. so that your $METAFLOW_ARGO_EVENTS_WEBHOOK_URL is a valid one.

yup! I was able to get it wired together - just had to change the namespace of my sensor! there are some other clunky things in my local environment setup w/ minikube…

Awesome!

Yea, i have a PR waiting to be merge to allow configuring the namespace for this

@billowy-salesmen-57704 Thank you for your PR!! I bumped it today to get some more traction. I need to make a similar to create a

METAFLOW_ARGO_WORKFLOWS_S3_ENDPOINT_URL

rather than using

METAFLOW_S3_ENDPOINT_URL

. this is specific for port forwarding a local environment deployed to something like minikube.