https://outerbounds.com/ logo
Join Slack
Powered by
<@U05FS3GELLQ> I have another issue maybe you coul...
# ask-metaflow
c
@User I have another issue maybe you could help me with. My autoscaler doesn't startup new nodes.
👀 1
✅ 1
I'm not sure I'm setting up the node groups correctly. I've added a more instance types will that allow me to ask for more CPU's? I'm very new to this world and would very much appreciate any help.
Copy code
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "17.23.0"

  cluster_name    = local.cluster_name
  cluster_version = "1.24"
  subnets         = module.vpc.private_subnets
  enable_irsa     = true
  tags            = local.tags

  vpc_id = module.vpc.vpc_id

  node_groups_defaults = {
    ami_type  = "AL2_x86_64"
    disk_size = 50
  }


  node_groups = {
    main = {
      desired_capacity = 1
      max_capacity     = 500
      min_capacity     = 1

      instance_types = ["r5.large", "c4.large", "c4.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge"]
      update_config = {
        max_unavailable_percentage = 50
      }
    }
  }

  workers_additional_policies = [
    aws_iam_policy.default_node.arn,
    aws_iam_policy.cluster_autoscaler.arn,
  ]
}
When I run a step with 
@kubernetes(cpu=8, memory="14000")
I get
Copy code
0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
h
This can be incorrect, but from what I understand about how the autoscaler handles the 
instance_types
field in the node group, it always selects the first instance type and only goes to the next if the first type is unavailable due to a quota limit or something. I am not sure if it actually decided to go down the order if the first instance type is too small to fit the requested resource. I'd create a node group per instance type and set the 
min
and 
desired
capacity to 0 for all but the r5.large one.
c
Thanks I'll try that. I assume I just copy paste the current node group and change the instance.
h
Yep, just need a different name/key for each entry in the node groups map.
c
So I created this
Copy code
node_groups = {
    main = {
      desired_capacity = 1
      max_capacity     = 5
      min_capacity     = 1

      instance_types = ["r5.large"]
      update_config = {
        max_unavailable_percentage = 50
      }
    }

    larger = {
      desired_capacity = 1
      max_capacity     = 5
      min_capacity     = 1

      instance_types = ["r5.xlarge"]
      update_config = {
        max_unavailable_percentage = 50
      }
    }
Then had 
@kubernetes(cpu=4, memory="6000")
which a r5.xlarge should have no trouble with. But I get
Copy code
0/2 nodes are available: 2 Insufficient cpu. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.
I also see this
Copy code
NotTriggerScaleUp	a few seconds ago	cluster-autoscaler	pod didn't trigger scale-up: 1 Insufficient ephemeral-storage, 2 Insufficient cpu
h
Can you try requesting 3 CPUs instead? It is likely that it is not allowing you to schedule a pod that requests 4 CPUs on a node with 4 CPUs because there are other things that need to run alongside the pod.
c
Doesn't seem to work
h
Can you share the 
requests
section of the pod that is in a pending state? Just want to confirm it is asking for 3vCPU and 6Gi memory
c
How do I do that?
h
You can run 
kubectl -n <your-namespace> get pod <pending-opod-name> -oyaml
. Then you can look for the 
resources
block in the yaml. Here is an example block:
Copy code
resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
c
Copy code
resources:
      requests:
        cpu: "3"
        ephemeral-storage: 10240M
        memory: 6G
h
My guess is that your disk size on the nodes is too small. I don't see you setting the 
disk_size
on your node groups. Are you setting them as part of a launch template somewhere? Can you check what the current size of the disk is?
c
Where can I find that? I've never setup a 
launch template
It appears to be set in 
<http://eks.tf|eks.tf>
Copy code
node_groups_defaults = {
    ami_type  = "AL2_x86_64"
    disk_size = 50
  }
h
Just to confirm, can you describe the 
r5.large
node in k8s. The ephermal storage size should be under the 
Allocatable
key. Since you are setting up the other node group the same way, I'd expect it to have the same disk size.
c
This is my eks.tf
Copy code
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "17.23.0"

  cluster_name    = local.cluster_name
  cluster_version = "1.24"
  subnets         = module.vpc.private_subnets
  enable_irsa     = true
  tags            = local.tags

  vpc_id = module.vpc.vpc_id

  node_groups_defaults = {
    ami_type  = "AL2_x86_64"
    disk_size = 50
  }


  #  "c4.large", "c4.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge"
  node_groups = {
    main = {
      desired_capacity = 1
      max_capacity     = 5
      min_capacity     = 1

      instance_types = ["r5.large"]
      update_config = {
        max_unavailable_percentage = 50
      }
    }

    larger = {
      desired_capacity = 0
      max_capacity     = 5
      min_capacity     = 0

      instance_types = ["r5.xlarge"]
      update_config = {
        max_unavailable_percentage = 50
      }
    }

    largerest = {
      desired_capacity = 0
      max_capacity     = 5
      min_capacity     = 0

      instance_types = ["r5.2xlarge"]
      update_config = {
        max_unavailable_percentage = 50
      }
    }

    superlargest = {
      desired_capacity = 0
      max_capacity     = 5
      min_capacity     = 0

      instance_types = ["r5.8xlarge"]
      update_config = {
        max_unavailable_percentage = 50
      }
    }


  }

  workers_additional_policies = [
    aws_iam_policy.default_node.arn,
    aws_iam_policy.cluster_autoscaler.arn,
  ]
}


resource "aws_iam_policy" "default_node" {
  name_prefix = "${local.cluster_name}-default"
  description = "Default policy for cluster ${module.eks.cluster_id}"
  policy      = data.aws_iam_policy_document.default_node.json
}

data "aws_iam_policy_document" "default_node" {
  statement {
    sid    = "S3"
    effect = "Allow"

    actions = [
      "s3:*",
      "kms:*",
    ]

    resources = ["*"]
  }
}

resource "aws_iam_policy" "cluster_autoscaler" {
  name_prefix = "cluster-autoscaler"
  description = "EKS cluster-autoscaler policy for cluster ${module.eks.cluster_id}"
  policy      = data.aws_iam_policy_document.cluster_autoscaler.json
}

data "aws_iam_policy_document" "cluster_autoscaler" {
  statement {
    sid    = "clusterAutoscalerAll"
    effect = "Allow"

    actions = [
      "autoscaling:DescribeAutoScalingGroups",
      "autoscaling:DescribeAutoScalingInstances",
      "autoscaling:DescribeLaunchConfigurations",
      "autoscaling:DescribeTags",
      "ec2:DescribeLaunchTemplateVersions",
    ]

    resources = ["*"]
  }

  statement {
    sid    = "clusterAutoscalerOwn"
    effect = "Allow"

    actions = [
      "autoscaling:SetDesiredCapacity",
      "autoscaling:TerminateInstanceInAutoScalingGroup",
      "autoscaling:UpdateAutoScalingGroup",
    ]

    resources = ["*"]

    condition {
      test     = "StringEquals"
      variable = "autoscaling:ResourceTag/kubernetes.io/cluster/${module.eks.cluster_id}"
      values   = ["owned"]
    }

    condition {
      test     = "StringEquals"
      variable = "autoscaling:ResourceTag/k8s.io/cluster-autoscaler/enabled"
      values   = ["true"]
    }
  }
}


data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

data "aws_caller_identity" "current" {}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}
@hundreds-zebra-57629 Any insights on the autoscaling. I just can't seem to get it to work.
h
Sorry for the delay. Based on the last error message you shared:
Copy code
NotTriggerScaleUp	a few seconds ago	cluster-autoscaler	pod didn't trigger scale-up: 1 Insufficient ephemeral-storage, 2 Insufficient cpu
My thought is that there is not enough disk space on the node.
Have you tried bumping up the disk size on the nodes?
c
I want the node to scale to 0 instances when not in use does that mean there is no disk space?
h
Nope, that is just a value set in the AWS autoscaling group so that by default it will always have zero nodes
The disk space is the one you shared in the link that is set to 50 by default
c
I'll give it a try
h
Also can you please share the output of:
Copy code
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{.status.allocatable}{"\n"}{end}'
c
ip-10-10-19-218.us-west-1.compute.internal{"cpu":"1930m","ephemeral-storage":"192188443124","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"15379388Ki","pods":"29"}
h
Humm, unless the 
r5.xlarge
is configured using a different value for disk, it looks like ephermal-storage shouldn't be an issue.
c
This is what I see from the autoscaler logs
Copy code
I0419 18:05:21.474205       1 orchestrator.go:542] Pod default/t-1336a11f-nf54f-d5n8l can't be scheduled on eks-mf-i7fc4m46-main20240419175615353800000003-b2c77bc0-3be9-62ab-a7d0-9db3ffed4b9c, predicate checking error: Insufficient cpu; predicateName=NodeResourcesFit; reasons: Insufficient cpu; debugInfo=
I0419 18:05:21.474257       1 orchestrator.go:542] Pod default/t-1336a11f-nf54f-d5n8l can't be scheduled on eks-mf-i7fc4m46-memory_instances20240419175615353800000001-12c77bc0-3be7-931b-4659-5f3ec089140f, predicate checking error: Insufficient ephemeral-storage; predicateName=NodeResourcesFit; reasons: Insufficient ephemeral-storage; debugInfo=
I0419 18:05:21.474274       1 orchestrator.go:150] No pod can fit to eks-mf-i7fc4m46-main20240419175615353800000003-b2c77bc0-3be9-62ab-a7d0-9db3ffed4b9c
I0419 18:05:21.474292       1 orchestrator.go:150] No pod can fit to eks-mf-i7fc4m46-memory_instances20240419175615353800000001-12c77bc0-3be7-931b-4659-5f3ec089140f
I0419 18:05:21.474308       1 orchestrator.go:164] No expansion options
I've configured them with 200GB of disk now
h
I'd have expected to see logs mentioning the 
large
node group as well. Do you see any?
c
I changed the name from before to memory_instances I thought maybe it was conflicting
Screenshot 2024-04-19 at 11.08.18.png
h
Updating here incase anyone else runs into this issue: Basically, when you set the min and desired count for node-group to 0, autoscaler doesn't really have a way of knowing that the size of the disk configured to the node-group. So it just ignores it because of lack of ephemeral-storage. The fix is to add an annotation tag to the autoscaling group to tell autoscaler that there is enough storage. You can add this terraform:
Copy code
resource "aws_autoscaling_group_tag" "cluster_autoscaler_resource_tags" {
  for_each               = module.eks.eks_managed_node_groups
  autoscaling_group_name = each.value.node_group_autoscaling_group_names[0]

  tag {
    key   = "<http://k8s.io/cluster-autoscaler/node-template/resources/ephemeral-storage|k8s.io/cluster-autoscaler/node-template/resources/ephemeral-storage>"
    value = "100G"
    propagate_at_launch = true
  }
}
m
I seem to have run into the same issue @hundreds-zebra-57629. The version of the EKS terraform module used in the original code is 17.23.0, which doesn't have the output 
eks_managed_node_groups
so this snippet above won't work with it. Is there an updated EKS configuration for a more recent module version somewhere that this snippet works with?
✅ 1
This is what I changed it to:
Copy code
resource "aws_autoscaling_group_tag" "cluster_autoscaler_resource_tags" {
  for_each               = module.eks.node_groups
  autoscaling_group_name = each.value.resources[0].autoscaling_groups[0].name

  tag {
    key   = "<http://k8s.io/cluster-autoscaler/node-template/resources/ephemeral-storage|k8s.io/cluster-autoscaler/node-template/resources/ephemeral-storage>"
    value = "50G"
    propagate_at_launch = true
  }
}
and it seems to have worked. Thanks!
✅ 1
among us party 1
m
It seems that I have an identical problem, but I use GCP, not AWS. Any suggestions on how to solve it on GCP would be appreciated.
h
Hey @melodic-ice-23640 can you please describe your setup in GCP? I don't recall seeing such a problem in a GCP setup.
m
@hundreds-zebra-57629 Thank you for answering. After further investigation, it seems that I had to increase disk size even further, to around 2.5x the amount required by the task, probably there is some limit how big part of a node's disk space can a single pod request from a node. Now it seem to work fine.
2 Views
Open in Slack
PreviousNext
Powered by