Hey Folks! Nice to meet you all. I am relatively new to working with Metaflow and was hoping to pick the collective brain. Currently running Metaflow with Argo Workflows on GCP GKE. We are seeing increasingly more occurrences of pods getting preempted as part of Autoscaler pod consolidation events. More often than not these are getting preempted before populating into Metaflow and so I thought to implement Argo Workflow retries to assist with but I am being told by the team, this is possibly not a great solution. I supposed closest issue I found was here. Have any of you ran into this on GKE and any suggestions for a more resilient deployment?

ideally, you would want to ensuer that the autoscaler allows these batch jobs to run till completion. retries and checkpointing can help but there isn't any reason to consolidate these pods very frequently

I'm not on GKE, but rather EKS with Karpenter. In our setup, we configure our metaflow pods to end up on nodes in a specific node pool that has consolidation set to only occur when nodes are empty. Maybe there is a similar option in your setup?

saw somewhere earlier this year that there's now early support for karpenter on GKE but prob not a viable option here since its early days metaflow exposes

METAFLOW_KUBERNETES_ANNOTATIONS

and you can set annotations in a dict in the kubernetes decorator, so could you set

<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>

here? since it seems like a fairly rare problem, wonder if your autoscaler is set to be overly aggressive so would look into that maybe first

We actually have kyverno policies already in place to deal with the safe-to-evict which has helped with other issues but does not safeguard against Preemptions from gke-balloon pods. The more I am reading into it, it seems like

balanced

presets might be the lever we have to pull here