hi, i'm trialing out Metaflow's --notify-on-success/--notify-on-error. On simulation of trigger this. The pod/tasks is stuck in pending state. Upon checking in kubernetes, to describe more details. We found that the "notification" pod seems to be not being able to be schedule due to toleration/taint that we've applied on our nodes. We do pass the toleration values via the "METAFLOW_KUBERNETES_TOLERATIONS": "[{\"key\":\"function\", \"operator\": \"Equal\", \"value\": \"cia-workloads\", \"effect\": \"NoSchedule\"}]", however, this doesn't seem to work. We did follow the additional instructions in the FAQ section on creating secrets against the service account used - https://github.com/Netflix/metaflow/releases/tag/2.9.1#features.1. However, it still fails in pending state. can i ask, are there specific ways to handle these, thanks ?

Interesting - let me check if the tolerations are supported for exit hooks

@square-wire-39606 Any updates on the above findings ?