Hey I have noticed in the MF UI the color bar that indicates the Run status is a bit buggy, it often shows Red that the run has failed but then a bit later changes back to Green when in reality the run has not failed. I've also noticed that the 'Finished at' column becaomes populated when it turns Red incorrectly and then goes blank when it changes back to green. Any ideas there?

are these scheduled runs on Step Functions / Argo Workflows? This misbehavior is a known bug related to how the status of prod deployments are shown

Daniel, this is a known problem which happens when metaflow fails to get a heartbeat. This can happen when things are busy. Is this your case?

Yes (Step Funcs). Cool. Things shouldn't be busy for us.

thanks for reporting this issue 👍 fixing it is not totally trivial since it involves querying the orchestrator status but it's on the roadmap

I am facing the same issue on Step Functions as well

good to know, thanks Yudhiesh

Is there something configurable for this? I'm noticing that basically all of my flows are Red (Failed) when they're really not. Of course, I have many flows running.

You can set the environment variable

HEARTBEAT_THRESHOLD

. By default it is

WAIT_TIME

* 6 = 60. Try increasing it until you no longer get the false negatives. See https://github.com/Netflix/metaflow-service/blob/master/services/ui_backend_service/docs/environment.md#heartbeat-intervals

@brave-lion-15961 I could try that. I'm using the outerbounds terraform module. Would that be added here? I'm not sure if that would go into

<https://registry.terraform.io/modules/outerbounds/metaflow/aws/latest#input_extra_ui_backend_env_vars|extra_ui_backend_env_vars>

or

<https://registry.terraform.io/modules/outerbounds/metaflow/aws/latest#input_extra_ui_static_env_vars|extra_ui_static_env_vars>

?

that would be

extra_ui_backend_env_vars

@brave-lion-15961 @average-beach-28850 is HEARTBEAT_THRESHOLD ms, s, or?

Seconds. As the default is 60, I would suggest doubling it to 120 and see how that goes.

It is a little better but I think what is happening is that I am launching many flows (manually triggering step) in a loop that end up getting queued up. Perhaps while they wait to eventually get executed metaflow ui thinks they're failed? BTW I increased to the heartbeat to 300.

I would recommend steadily increasing it until you no longer see the problem - I would be interested to know what that number is. The downside of a larger number is that it will take longer to know if a task really failed

Yeah - we use a special pattern for our use case where we trigger many of the same flow (diff parameters). I set it to 300 and still observe pretty much everything red until later on when it basically all turns green. We use AWS Step/Batch. I wonder what the deal here is because I'd assume at Netflix there were many flows running all at once. Perhaps this pattern causes some issue.