Hi Outerbounds, I hope you are well. I'm currently working on setting up alerting for our Metaflow runs and came across the following releases: • Metaflow 2.9.10 • Metaflow 2.9.1 I noticed the environment variables

METAFLOW_ARGO_WORKFLOWS_CREATE_NOTIFY_PAGER_DUTY_INTEGRATION_KEY

and

METAFLOW_ARGO_WORKFLOWS_CREATE_NOTIFY_SLACK_WEBHOOK_URL

mentioned in these releases. However, I couldn't find any references to these variables in the Metaflow repository. Could you please provide more details or documentation on how to use these variables in our workflows? Specifically, I would like to understand: • How to configure these environment variables. • How these variables integrate with Metaflow and Argo Workflows. • Any examples or best practices for setting up notifications using these variables. Any help or guidance would be greatly appreciated. Thank you!

Hi! These environment variables are generated by Metaflow behind the scenes. You can configure these in your bashrc for example

Hi @ancient-application-36103, thanks for getting back to me! I have another question, it mentions that we can specify those environment variables, then how do we set up the --notify-on-error for pagerduty, when I tried that it still asks for the pagerduty_integration_key in the cli

can you share the terminal logs and the command that you executed?

This is the terminal I tried to run

python flow.py --package-suffixes=.json,.txt,.sql --environment=pypi argo-workflows create --notify-on-error

and I set up the env variables in our packages, this is added in the

mfextinit_{company_name}.py

# Alerts Set Up
METAFLOW_ARGO_WORKFLOWS_CREATE_NOTIFY_PAGER_DUTY_INTEGRATION_KEY = cia_config.PAGERDUTY_INTERGRATION_KEY
METAFLOW_ARGO_WORKFLOWS_CREATE_NOTIFY_SLACK_WEBHOOK_URL = cia_config.SLACK_WEBHOOK_URL

And the error I found was

Flow failed:
    Notifications require specifying an incoming Slack webhook url via --notify-slack-webhook-url or PagerDuty events v2 integration key via --notify-pager-duty-integration-key.
     If you would like to set up notifications for your Slack workspace, follow the instructions at <https://api.slack.com/messaging/webhooks> to generate a webhook url.
     For notifications through PagerDuty, generate an integration key by following the instructions at <https://support.pagerduty.com/docs/services-and-integrations#create-a-generic-events-api-integration>

Thanks!

how do the entries from

mfextinit_{company_name}.py

get to the environment?

Oh yeap that worked! Thanks Savin! One question, is there a way we can customise the alerts messages?

Not yet, but it is on our roadmap to expose more information. Any insights on what customization you are looking for?

Hey Savin, it would be nice if we can log the actual reason why the flow failed so it will be easier for the devs to look into the flow and figure out where it failed.

makes sense - i opened a GH issue to track this - https://github.com/Netflix/metaflow/issues/1888

Thanks Savin!