I noticed an interesting behavior where the sync between metaflow UI and argo workflow is broken for a long waiting task, and I am not sure if it's going to happen frequently moving forward. I have a long running task on argo - think 10 hours - which doesn't log anything to console (it logs into a file). So, on the console inside the container in argo you'll see something like this -

Waiting for magic...

The Metaflow UI also show the same log - great, as expected. BUT sometime after the launch (I was AFK but checked after 2.5 hours), metaflow UI showed the task as failed (at 17 minutes). But the argo workflow was still going strong (took about 6 hours). When the argo workflow finally completed, it exited with status 0 - no problems. I have other tasks that run for 1.5 hours, and I didn't see this error consistently (though I do recall a similar disconnect) What might be going on here?