https://outerbounds.com/ logo
Join Slack
Powered by
Has anyone seen metaflow and argo pods get deleted...
# ask-metaflow
e
Has anyone seen metaflow and argo pods get deleted during scale down events on kubernetes? It seems that when the autoscaler is trying to scale down, it will randomly pick nodes for scale down. If there happen to be running metaflow/argo pods on that node, then the pod gets terminated and the job fails. We've tried adding the annotation 
<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: false
to the argo pods but that doesn't prevent the issue
✅ 1
a
@User any thoughts here?
u
Hey Tyler... it's true that if a particular nodes' resource utilization is < 50% (or whatever is configured in autoscaler), it could terminate the node and cause disruption of the tasks. It's unlikely that these nodes are picked at random though. See this for how the autoscaler decides to terminate nodes. Adding the annotation should help though. If it's okay to have the tasks retried, you could use Metaflows @retry decorator so that the tasks get retried automatically on failures. Are you running on any public cloud? Also, is there a specific scenario when you see these failures (e.g. when there is a large 
foreach
)
m
We had this before (unrelated to Metaflow) and in our case it was due to the kubernetes cluster rebalancing the nodes between regions. This is the default behaviour for EKS to met HA requirements. Perhaps this is the issue you are having? I believe in this case annotations will make no difference to the eviction behaviour. In our case we don't have any HA requirements, so made the separate autoscaling groups for each region.
u
+1. We've seen failures when AWS rebalances nodes across availability zones. This can happen especially when a large number of pods complete and cluster autoscaler terminates some nodes because they are underutilized. In these cases, AWS terminates some and starts new nodes to maintain near equal nodes across AZs.
u
For AWS, you could disable the 
AZRebalance
option on the auto-scaling group: https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html
e
Thanks for the replies
is there a specific scenario when you see these failures
We've seen this when one job is long running, and then another job is kicked off which causes the cluster to scale up. If the 2nd job completes and the cluster is now in need to scale down, occasionally it will pick the node that the first job is running on to terminate thus causing it to fail.
Are you running on any public cloud?
Yes, this is on AWS using EKS Really appreciate the tips all. I'll test out suspending the AZRebalance process and report back
u
Cool. This is very likely because of the AZRebalance problem. You can look at the ASG's 
Activity
tab in the AWS console to see if AWS terminated some nodes for rebalancing zones.
e
Turns out this was due to the 
--balance-similar-node-groups
flag on the autoscaler being enabled. We deployed our EKS cluster using an internally customized version of the terraform aws eks module, and it had this flag hardcoded in it. Thanks again for the direction
u
Awesome! Glad this worked out.
e
@proud-eye-90172 Unfortunately I'm still having the same problem with metaflow jobs being marked for deletion during scale down events. I was able to get the following from the kubernetes events:
Copy code
default       41m         Normal    ScaleDown                node/ip-10-100-213-20.ec2.internal                       marked the node as toBeDeleted/unschedulable
kube-system   41m         Normal    ScaleDown                configmap/cluster-autoscaler-status                      Scale-down: removing node ip-10-100-213-20.ec2.internal, utilization: {0.458712715855573 0.585732366900626 0 memory 0.585732366900626}, pods to reschedule: kube-system/overprovisioning-95bf6758f-m76xf ...* argo/t-f8rp5-swbqx ...*
argo          41m         Normal    ScaleDown                pod/t-f8rp5-swbqx                                        deleting pod for node scale down
(* There were several copies of the overprovisioning pod that I truncated) And then this is from the autoscaling group:
Copy code
At 2023-04-24T21:43:32Z instance i-001f39cb3f5de5e55 was taken out of service in response to a user request, shrinking the capacity from 4 to 3
This is with the AZRebalance process disabled on all 3 nodegroups we have running Any thoughts of where to look next?
Here's our autoscaler config if that's helpful:
Copy code
resource "helm_release" "cluster_autoscaler" {
  name       = "cluster-autoscaler"
  repository = "<https://kubernetes.github.io/autoscaler/>"
  chart      = "cluster-autoscaler"
  namespace  = var.namespace
  version    = var.chart_version
  timeout    = 600

  provider = helm

  set {
    name  = "image.tag"
    value = "v1.23.0"
  }

  set {
    name  = "autoDiscovery.enabled"
    value = "true"
  }

  set {
    name  = "autoDiscovery.clusterName"
    value = var.cluster_name
  }

  set {
    name  = "autoDiscovery.tags"
    value = "{${join(",", ["<http://k8s.io/cluster-autoscaler/enabled=true|k8s.io/cluster-autoscaler/enabled=true>", "<http://k8s.io/cluster-autoscaler/${var.cluster_name}=owned|k8s.io/cluster-autoscaler/${var.cluster_name}=owned>"])}}"
  }

  set {
    name  = "cloudProvider"
    value = "aws"
  }

  set {
    name  = "awsRegion"
    value = var.region
  }

  set {
    name  = "rbac.create"
    value = "true"
  }

  set {
    name  = "rbac.serviceAccount.name"
    value = var.service_account
  }

  set {
    name  = "rbac.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = module.iam_assumable_role_admin.iam_role_arn
  }

  set {
    name  = "extraArgs.expander"
    value = local.ca_expander
  }

  set {
    name  = "expanderPriorities"
    value = local.expander_config
  }

  set {
    name  = "extraArgs.balance-similar-node-groups"
    value = "false"
  }

  set {
    name  = "extraArgs.balancing-ignore-label_1"
    value = "<http://topology.ebs.csi.aws.com/zone|topology.ebs.csi.aws.com/zone>"
  }

  set {
    name  = "extraArgs.balancing-ignore-label_2"
    value = "<http://vpc.amazonaws.com/eniConfig|vpc.amazonaws.com/eniConfig>"
  }

  set {
    name  = "extraArgs.balancing-ignore-label_3"
    value = "<http://beta.kubernetes.io/instance-type|beta.kubernetes.io/instance-type>"
  }

  set {
    name  = "extraArgs.balancing-ignore-label_4"
    value = "<http://node.kubernetes.io/instance-type|node.kubernetes.io/instance-type>"
  }

  set {
    name  = "extraArgs.balancing-ignore-label_5"
    value = "<http://eks.amazonaws.com/sourceLaunchTemplateId|eks.amazonaws.com/sourceLaunchTemplateId>"
  }

  set {
    name  = "extraArgs.balancing-ignore-label_6"
    value = "<http://eks.amazonaws.com/sourceLaunchTemplateVersion|eks.amazonaws.com/sourceLaunchTemplateVersion>"
  }

  set {
    name  = "extraArgs.balancing-ignore-label_7"
    value = "<http://eks.amazonaws.com/capacityType|eks.amazonaws.com/capacityType>"
  }

  set {
    name  = "extraArgs.skip-nodes-with-system-pods"
    value = "false"
  }

  set {
    name  = "extraArgs.scale-down-unneeded-time"
    value = "5m0s"
  }

  set {
    name  = "extraArgs.scale-down-unready-time"
    value = "10m0s"
  }
  # Reduce provision time 15->5 to fallback to ondemand quickly
  set {
    name  = "extraArgs.max-node-provision-time"
    value = "5m0s"
  }

  set {
    name  = "extraArgs.scale-down-utilization-threshold"
    value = "0.75"
  }

  set {
    name  = "extraArgs.skip-nodes-with-local-storage"
    value = "false"
}
My current suspicion is that our companies autoscaler is set too aggressively with the 
0.75
utilization threshold and the 5 min unneeded time
u
Looking into this ...
p
Yeah.. tweaking the 
scaleDownUtilizationThreshold
option might be a place to start. You can see the actual utilization in that autoscaler log output. So, keeping the threshold below that might be a good experiment. But, it will affect other autoscaling behavior and maybe increase costs. One other option is to add a small piece of code in the step to disable autoscaler from scaling down a node on which the step is running. You can add the following annotation to the node:
Copy code
<http://cluster-autoscaler.kubernetes.io/scale-down-disabled|cluster-autoscaler.kubernetes.io/scale-down-disabled>: true
e
Is there a way to add an annotation to the pods for metaflow itself? This one disabled autoscaling at the pod spec level rather than the node level: 
"<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>": "false"
a
You can use OPA to attach an annotation to all pods
m
If it is the autoscaler, you should be able to see the scale down events in the logs. Probably worth checking these logs to confirm this is the issue.
It will also contain node utilisation stats and so on, so you can see the reason a node was scaled down
e
@mammoth-rainbow-82717 I think I've determined that it is due to utilization threshold based on this event:
Copy code
kube-system   41m         Normal    ScaleDown                configmap/cluster-autoscaler-status                      Scale-down: removing node ip-10-100-213-20.ec2.internal, utilization: {0.458712715855573 0.585732366900626 0 memory 0.585732366900626}
Our threshold is pretty high (0.75) so I think that's the reason for this. @square-wire-39606 Would it make sense to add this annotation to all jobs created by metaflow by adding the below? I can't think of a use case where it would be useful for a running metaflow job to be targeted for scale down by the autoscaler:
Copy code
job.annotation("<http://autoscaler.kubernetes.io/safe-to-evict|autoscaler.kubernetes.io/safe-to-evict>", "false"),
https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/kubernetes/kubernetes.py#L272
m
OK, coo. Yeah, this is the type of thing you'll see in the logs. 0.75 does seem pretty aggressive fwiw 🙂
💯 1
u
@elegant-beach-10818 To test whether the autoscaler config is the real problem, can you run the flow by simply increasing the resources allocated to the pod ( increase CPU or memory in 
@resources
or 
@kubernetes
). We can ensure that if the pod had resources > autoscaler threshold, the flows complete. Then, it could come down to adding the annotation or changing the autoscaler config.
u
We've tried adding the annotation 
<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: false
to the argo pods but that doesn't prevent the issue
The annotation should go on the node, not on the pod, right? Nevermind: The annotation is expected to be set on the pod.
e
@proud-eye-90172
We've tried adding the annotation 
<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: false
to the argo pods but that doesn't prevent the issue
This was before I disabled the AZRebalance process in the autoscaling groups. I'm pretty sure the AZRebalance doesn't respect pod annotations I was able to reproduce the error with the following setup: 2 nodegroups that both only have the 6mi.4xlarge instances (both nodegroups have the AZRebalance process disabled) Used the decorator 
@kubernetes(cpu=2, memory=33000)
on a foreach step that would create 4 children steps. Because the instance has 64 GB of RAM, it wouldn't be possible to schedule multiple foreach steps on the same Node. Ran this flow 4 times in parallel so that there were 16 total nodes. After all the pods had been scheduled, I killed 2 of the flows so there are only 8 steps running. After a 5 minutes one of the remaining 8 steps failed and I observed this kubernetes event:
Copy code
kube-system   43m         Normal    ScaleDown                      configmap/cluster-autoscaler-status                   Scale-down: removing node ip-10-100-181-71.ec2.internal, utilization: {0.3530522341095028 0.6419678241519629 0 memory 0.6419678241519629}, pods to reschedule: argo/t-2w9zh-2tkb ...
followed by this
Copy code
argo          43m         Warning   BackoffLimitExceeded           job/t-2w9zh                                           Job has reached the specified backoff limit
It seems that it is indeed related to the autoscaler In a 2nd test, I added the annotation 
.annotation("<http://autoscaler.kubernetes.io/safe-to-evict|autoscaler.kubernetes.io/safe-to-evict>", "false")
to this file: https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/kubernetes/kubernetes.py#L273 I then pip installed this fork of metaflow. I ran the same test as above and didn't have any of the remaining 8 steps fail after killing 2 flows. I also noticed this in the autoscaler logs:
Copy code
I0425 17:03:32.707952       1 cluster.go:192] Fast evaluation: node ip-10-100-181-254.ec2.internal is not suitable for removal: failed to find place for argo/t-8zwsn-gkgxh
It seems that the annotation successfully prevented the autoscaler from trying to reschedule the pod
u
This was an excellent exercise! Great work! I agree that setting the annotation by default would be a good idea. Can you create an issue on Github and send out a PR? We can get more feedback on this setting from the community. There might be some objections to this, especially if someone is running with 
@retry
and are okay with their pods restarting on a different node in order to save costs. But, it'll be best to ask.
🙌 1
e
Looks like I had a typo with the original annotation. Fixing that shows logs like below in the autoscaler:
Copy code
I0425 21:25:38.435423       1 cluster.go:169] Fast evaluation: node ip-10-100-177-47.ec2.internal cannot be removed: pod annotated as not safe to evict present: t-r5qgg-fgjv7
7 Views
Open in Slack
PreviousNext
Powered by