nutritious-forest-60017
03/27/2026, 8:55 PMself.next({...}, condition=...)) on recent Argo Workflows versions. Flows hang at the convergence step after conditional branches. The pod is never scheduled. I tested this out with the metaflow-music-agent example you all put together using claude code to help me comb through my cluster, metaflow source code, and argo source code to find the issue.
What happens: The generated DAG template for the convergence step (e.g. agent_complete) references {{tasks.done.outputs.parameters.task-id}} ,{{tasks.failed.outputs.parameters.task-id}} , etc. in its input-paths parameter. Only one branch runs; the others are skipped. The conditional_input_paths.py runtime filter is supposed to strip out the unresolved paths, but the pod never starts so it never runs.
Why: Argo PR argoproj/argo-workflows#15442 (merged Feb 2026) changed template.Replace(allowUnresolved=true) to template.ReplaceStrict in dag.go. Unresolved tasks.* references now cause an infinite requeue instead of being passed through as literal strings. This was cherry-picked to both v3.7.11 and v4.0.2.
Argo side: This is being tracked in argoproj/argo-workflows#15737. PR #15736 has been merged and cherry-picked to v4.0 and v3.7 but that fix targets expanded task properties (`withItems`/`withParams`), so it may not fully cover the skipped conditional branch case. Worth verifying once v3.7.12 / v4.0.4 are released.
Affected: Argo v3.7.11+, v4.0.2+
Works on: Argo v3.7.10, v4.0.1 and earlier
Tested with: Metaflow 2.19.22, 2.19.0, reproduced using the metaflow-music-agent example (https://github.com/outerbounds/metaflow-music-agent)
Just wanted to make you all aware of this even if it ends up getting fixed on Argo's end.ancient-application-36103
03/27/2026, 8:57 PMancient-application-36103
03/27/2026, 8:57 PMhundreds-rainbow-67050
03/27/2026, 9:16 PMnutritious-forest-60017
03/27/2026, 9:20 PM