I have a question about a recent change to the argo workflows labels. It seems like you have removed adding labels to the

podMetadata

section of the workflow template? Is there a reason for this? We were actually using those in a custom onExit handler. If would be nice to have them there, but if that was causing issues I have another workaround(s) in mind. Curious to know if there are side effects to having these, because one of my workarounds would be to simple add them back to this section with the mutating webhook that we use to add the onExit handler. The other would be to add them to the handler itself.

hi! @thankful-ambulance-42457 may have the context

Forgot the link, the change is here: https://github.com/Netflix/metaflow/pull/1568/files#r1915256238

oh sorry, didn't notice the thread here. Already responded some on GH, but lets continue here 🙂

Basically, with a workflow created with 2.12(.38), we see the labels in

podMetadata.labels

but in a flow created with 2.13(.2), we see the labels repeated as part of each task template. For normal metaflow usage this seems functionally the same, but any other tasks don't inherit these labels. Which could be the intended functionality

so in a nutshell, conceptually it made the most sense that any labels coming from

@kubernetes

only apply to the task pods, nothing else (on argo). We can introduce separate env vars for argo to extend the other bits, such as exit handlers etc.

I do think that our case is very niche. I could see a case where people would want all tasks to have a certain set of labels and then have additional labels for certain tasks. This can probably be done pretty easily with python tho.

now for the argo extensibility, I haven't had many thoughts on the matter yet, but at the very least some

ARGO_SHARED_LABELS/ARGO_SHARED_ANNOTATIONS

which would apply to all argo resources might be justified.

Cool, good to know. I think for now, we will grab the labels from a known task like

start

and add them to our onExit template ourselves.

cheers, good to hear you're not blocked with the changes, and apologies for the extra work with the unintentional breaking change 😅

no worries, it's only slack error messages flooding our channel 🙂 nothing too major