Another question - we semi-frequently see Metaflow runs get created but the corresponding run is not present in the Argo Workflows UI, has anyone else hit this / know where to look to try to debug this issue?

that's weird. are you talking about runs that are created from the workflow template that's deployed to argo or just

python flow.py run

- the latter won't show up because the flow is nopt being executed using argo

runs created from the workflow template deployed to argo

if you end up garbage collecting them from etcd, that could be one of the reasons.

hmmm, do you also run with an external postgres for the persistent storage or is etcd your persistent storage?

We have both modes

Interesting - ill keep looking, but it was some weird behavior we saw