https://outerbounds.com/ logo
Join Slack
Powered by
anyone using metaflow on gcp with argo events? I'v...
# ask-metaflow
g
anyone using metaflow on gcp with argo events? I've configured a flow with @trigger(
date_updated
) and am trying to trigger it via Jupyter notebook -
✅ 1
metaflowconfig looks like this
Copy code
{
  "METAFLOW_DATASTORE_SYSROOT_GS": "<gs://bucket-name/bucket-root>",
  "METAFLOW_DEFAULT_DATASTORE": "gs",
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_KUBERNETES_NAMESPACE": "metaflow",
  "METAFLOW_KUBERNETES_SERVICE_ACCOUNT": "metaflow-service",
  "METAFLOW_SERVICE_INTERNAL_URL": "<http://metaflow-service.metaflow:8080/>",
  "METAFLOW_SERVICE_URL": "<http://127.0.0.1:8080/>",
  "METAFLOW_DEFAULT_SECRETS_BACKEND_TYPE": "gcp-secret-manager",
  "METAFLOW_ARGO_EVENTS_EVENT": "metaflow-event",
  "METAFLOW_ARGO_EVENTS_EVENT_BUS": "default",
  "METAFLOW_ARGO_EVENTS_EVENT_SOURCE": "argo-events-webhook",
  "METAFLOW_ARGO_EVENTS_SERVICE_ACCOUNT": "operate-workflow-sa",
  "METAFLOW_ARGO_EVENTS_WEBHOOK_URL": "<http://argo-events-webhook-eventsource-svc.argo-events:12000/metaflow-event>"
}
and I have my ports forwarded. the argo setup was adapted from from the terraform for gcp here
a
the webhook url as configured is only accessible from within the kubernetes cluster. is your jupyter notebook also running there?
g
oops, yeah I updated that to localhost. however it didn't work until I added the following to metaflow/plugins/argo/argo_events.py
Copy code
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
more general question I guess, what is the recommended pattern of deploying argo-events/argo-workflows + metaflow I have metaflow services deployed to the 
metaflow
ns, argo-events and argo-workflows to their own respective namespaces. I would like all metaflow-related pods to be launched within 
metaflow
namespace. when i create my workflow:
Copy code
python3 workflows/event_triggered_flow.py --environment=conda argo-workflows create
it gets created in argo under 
metaflow
ns because of the metaflowconfig. but when I run a jupyter notebook cell (see above) the WF in metaflow does not get triggered, but a workflow I previously created with 
METAFLOW_KUBERNETES_NAMESPACE=argo-events
does. does this mean I have to put everything in one ns? I saw this thread and I guess I'm still wondering what the path forward is
b
https://github.com/Netflix/metaflow/pull/1463 I have been waiting on this for a while. We deploy all the services to one namespace, but the pods run in other namespaces. I think you could put them all in different namespace as long as you have the internal URLs correctly configured.
👀 1
👍 1
g
^^ that would be nice! hope to see it merged soon
a
@ancient-application-36103 @victorious-lawyer-58417 Any timeline for when this PR could be merged in? We have a workaround but having a separate namespace for argo events declared in the metaflow config would be helpful.
a
At the moment, we are looking into setting up a test harness to be able to ensure that there are no regressions with changes of this nature. Once that lands we should be in a good position to merge this in
thankyou 1
b
We ended up just creating one jetstream deployment per namespace and abandoning this PR.
It probably only needs a few tweaks to work tho. The argo workflows code has changed a little but not that much.
18 Views
Open in Slack
PreviousNext
Powered by