I'm trying to understand the

argo-workflows --enable-heartbeat-daemon

feature. 1. What does the heartbeat daemon do? Does it monitor the flow for status, being more reliable and robust to K8s errors that may arise during pod initialization of a task pod? Or does is it more outward focused, reporting task status based on the argo template configuration of heartbeat task and step task operating at the same time? 2. When is it a good idea to use the heartbeat daemon? 3. When is it not a good idea to use it?

Heya, sorry for the long wait 😓 The heartbeat daemon is a standalone process in the argo workflow that is responsible for ticking heartbeats on a run level, keeping the run status as

running

whenever the workflow is still active. The original use-case for this was to improve Metaflow UI run status accuracy in cases of very long scheduler delays between launching tasks, as without the daemon, the only source of heartbeats when running on argo workflows are the metaflow runtime within the task pods. This can lead to issues if between tasks finishing, and new tasks launching there is a long enough delay to make the heartbeats stale -> a run can incorrectly show as failed until a new task launches. Unfortunately the daemon approach to solving this issue had some major drawbacks due to the way argo daemons are handled. In some environments they simply cannot be used, which is why this feature defaults to disabled.