Hello, I am using metaflow with k8s. One of my flows has a foreach that spins +500 pods (I think 16 at time). I can see that the pods that are completed/failed are not being deleted. Some of them are shown as completed for days. When I inspect my nodes they have hundreds of pods in a completed/failed status. Is this a normal/expected behaviour?

Hi Alexander, yes, the standard TTL (Time-To-Live) for all pods (completed, failed) is 7 days

Thanks... given that those pods are still "alive" for 7 days, are they still consuming resources?... if, lets say, each of the pods is requesting 4 gigabytes of memory and 1 CPU and there are 500 pods, then those resources would be unavailable for other pods?

Once the pod is completed/failed, it does not take up any resources (CPU, memory…etc). However its worth noting that these completed pods still exist in etcd, which might result in some lag if you’re in the range of 15000+ pods. For example, if you’re using a CLI tool like k9s, you will notice some lag in load times. This does not impact the performance of your running pods though.

Got it, thanks a lot 🙏