Hi, I tried to activate slack notification but I s...
# ask-metaflow
f
Hi, I tried to activate slack notification but I see pods related to notifications (.e.g
onExit
or
on-failure
)were in pending state and then I checked the log. More details in the 🧵
1
Copy code
time="2023-06-06T14:07:04.607Z" level=info msg="Starting Workflow Executor" version=v3.4.0
time="2023-06-06T14:07:04.608Z" level=info msg="Starting Agent" requeueTime=10s taskWorkers=16 workflow=backfillwidthheight-k9p47
time="2023-06-06T14:07:04.615Z" level=info msg="Watch workflowtasksets 403"
Error: unknown (get <http://workflowtasksets.argoproj.io|workflowtasksets.argoproj.io>)
Usage:
  argoexec agent main [flags]

Flags:
  -h, --help   help for main

Global Flags:
      --as string                      Username to impersonate for the operation
      --as-group stringArray           Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
      --as-uid string                  UID to impersonate for the operation
      --certificate-authority string   Path to a cert file for the certificate authority
      --client-certificate string      Path to a client certificate file for TLS
      --client-key string              Path to a client key file for TLS
      --cluster string                 The name of the kubeconfig cluster to use
      --context string                 The name of the kubeconfig context to use
      --gloglevel int                  Set the glog logging level
      --insecure-skip-tls-verify       If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
      --kubeconfig string              Path to a kube config. Only required if out-of-cluster
      --log-format string              The formatter to use for logs. One of: text|json (default "text")
      --loglevel string                Set the logging level. One of: debug|info|warn|error (default "info")
  -n, --namespace string               If present, the namespace scope for this CLI request
      --password string                Password for basic authentication to the API server
      --proxy-url string               If provided, this URL will be used to connect via proxy
      --request-timeout string         The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
      --server string                  The address and port of the Kubernetes API server
      --tls-server-name string         If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.
      --token string                   Bearer token for authentication to the API server
      --user string                    The name of the kubeconfig user to use
      --username string                Username for basic authentication to the API server

unknown (get <http://workflowtasksets.argoproj.io|workflowtasksets.argoproj.io>)
The associated pod has the suffix
1340600742-agent
Not sure if server/client kubectl version mismatch can cause such issue or not.
Copy code
~ ❯❯❯ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: <http://version.Info|version.Info>{Major:"1", Minor:"25", GitVersion:"v1.25.9", GitCommit:"a1a87a0a2bcd605820920c6b0e618a8ab7d117d4", GitTreeState:"clean", BuildDate:"2023-04-12T12:16:51Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: <http://version.Info|version.Info>{Major:"1", Minor:"21+", GitVersion:"v1.21.14-eks-0a21954", GitCommit:"49b0defc3b6cc0a37d22817956669154c27f5525", GitTreeState:"clean", BuildDate:"2023-04-15T00:29:43Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.25) and server (1.21) exceeds the supported minor version skew of +/-1
s
f
I tried following the instructions provided in the link Savin but it doesn't fix it. Also the k8s we are running is 1.22 I only see these two deployments related to argo
Copy code
~ ❯❯❯ kubectl get deployments -n argo
NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
argo-argo-workflows-server                1/1     1            1           257d
argo-argo-workflows-workflow-controller   1/1     1            1           257d
When I see the argo workflows related to the flow which should send notification, I see these two notifications related step in PENDING
Copy code
NAME:flow-gqb5h.hooks.notify-on-failure   
ID: flow-gqb5h-2770537874   
TYPE: HTTP
PHASE: Pending
Copy code
NAME: flow-gqb5h.onExit   
ID: flow-gqb5h-2581194255   
TYPE: HTTP
PHASE: Pending
u
I suspect that the serviceaccount running this pod has a role that does not have adequate privileges to get the
<http://workflowtasksets.argoproj.io|workflowtasksets.argoproj.io>
objects. Can you check that role for that serviceaccount?
u
Hey @fast-vr-44972.. just wanted to follow-up regarding this. Were you able to make progress with this?
f
Hey Shri, I haven't been able to try it out. I will update it here once I get some time to try it out. Appreciate the help
👍 1
So this issue is kind of also blocked by the issue mentioned here since I think the notification has been added from some 2.9.x version. For avoiding that bug we are only using 2.8.x in our production currently.
a
Is --notify-on-error available for AWS batch workflows? I am reading these docs linked in the thread above by @User.
1
a
Only on Argo Workflows for now @adventurous-plastic-90730
👍 1