Hi, our argo workflow controller pod seems to be restarting quite often suddenly. In the log I see it fails liveness probe health check. Any idea how to fix it? I can't exactly figure out if it could be because of running out of resource or not.

Warning  Unhealthy  2m35s (x3 over 4m35s)  kubelet            Liveness probe failed: Get "<http://10.10.29.141:6060/healthz>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    2m35s                  kubelet            Container controller failed liveness probe, will be restarted

this can happen for a variety of reasons. do you have any telemetry on pod metrics?

sadly no

it's hard to pin point the issue otherwise unfortunately. how many executions (workflow objects) do you have currently in your deployment?

around 20! So what seem to have worked is to increase the timeout for liveness probe check from 30 seconds to 45 seconds. But yeah we don't know the root cause.

Are 20 the deployed flows (workflow templates in argo parlance) or executions of the deployed flows (workflows in argo parlance)?

20 is the deployed ones. Although we realized issue with the controller when some of our cron workflow didn't run last night. Then we saw lot of restart associated with the controller.

If you have too many stale workflow objects, that might result in the controller’s liveness endpoint slowing down to a crawl - its a conjecture - may or may not apply in your case

By workflow objects do you mean the execution or the deployed flows?

The executions

yeah those can be quite a lot.

the way the liveness probe for argo-workflows controller works is - it lists out all the workflow objects ( execution objects ) . If this is quite a lot then its possible that the liveness probe will not complete and fail. On breaching the liveness probe threshold the pod will restart. You can do 1 of the two things: • Increase the liveness probe timeout ( as you have done ) • [If possible] reduce the number of workflow objects from etcD so the listing of the workflows objects completes and subsequently the liveness probe.

By the way is there a general cleaning up mechanism? Like deleting say older than x number of days/months executions?

in metaflow? no.. you will have to use k8s APIs to write up something custom afaik