deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

EKS 1.32 upgrade: Karpenter 0.37 stops at Kubernetes 1.30, so upgrade order matters

A dev.to walkthrough shows how a team upgraded EKS 1.32 to 1.36 in place, replacing Karpenter 0.37 with 1.14.3 and AL2 nodes with AL2023 before touching the control plane, and lists eight pitfalls.

EKS 1.32 upgrade: Karpenter 0.37 stops at Kubernetes 1.30, so upgrade order matters

What the compatibility error actually means

When Karpenter 0.37 runs against a Kubernetes 1.32 cluster, it logs "failed validating kubernetes version" with the note that the Karpenter version is not compatible with K8s version 1.32. According to the write-up on dev.to, that line deserves a careful reading: version 0.37.x supports Kubernetes 1.23 through 1.30, the controller logs the mismatch once per start when it discovers the cluster version, and then carries on working. The audit behind the article found Karpenter had created seven nodes in the previous 24 hours despite the error, and the EC2NodeClass stayed Ready. Nothing is down because of that log line.

The per-start nature also makes log counts misleading: one of the team's two controller pods showed zero error lines over 72 hours, and the team only observed the message directly after a 0.37.8 pod started.

Why the cluster had to move anyway

An audit on 2026-11-14 framed the real problem: upstream patches for Kubernetes 1.32 ended on 2026-02-28 and EKS standard support ended on 2026-03-23, leaving the cluster on extended support. The control plane ran v1.32.13-eks-577a51e with nine worker nodes on v1.32.9, and the managed node group AMI was 341 days old.

The second deadline was the node operating system. EKS stopped publishing Amazon Linux 2 AMIs on November 26, 2025, and the eks list-insights output flagged an "Amazon Linux 2 compatibility" error for 1.33. Nine of the cluster's eleven nodes were Karpenter-created AL2 nodes, backed by an EC2NodeClass using the default AL2 AMI family with no pinned version.

The order of work

The recommended sequence, which the team followed on 2026-11-16: upgrade Karpenter first, switch the EC2NodeClass to AL2023 with a pinned AMI version, and only then update the control plane. The cluster, around 11 nodes declared in one Terraform repository and applied through GitHub Actions, was taken to EKS 1.36 in place, with Karpenter ending at 1.14.3.

One wrinkle shaped the first incident: the Karpenter install was chart 0.37.0 with two replicas, the webhook disabled, and no separate CRD release, so the nodepool, nodeclaim and ec2nodeclass CRDs served only v1beta1 with no conversion strategy.

Eight problems along the way

  1. The first Terraform plan after changing manifests from karpenter.sh/v1beta1 to karpenter.sh/v1 marked all five manifest resources for replacement: five to add, five to destroy. The team did not apply it.
  2. One second after an apply did go through, Karpenter marked all eight existing NodeClaims as drifted and began replacing them, despite the pull request claiming nodes would not change. Self-inflicted.
  3. Two nodes stayed on AL2 for 50 minutes because three pods carried the karpenter.sh/do-not-disrupt annotation and blocked disruption.
  4. During the 1.33 control-plane update, image pulls hit a QPS limit: 27 pods were not Running, 11 load-balancer target groups had no healthy target, and the cluster owner reported 503s. The write-up caveats that the 503s were not measured from outside. Most pods were Running again within about four minutes.
  5. Bitnami Redis images from public.ecr.aws failed with NotFound errors and sat in ImagePullBackOff.
  6. EKS refused the 1.34 update with a 400 error: each availability zone needs a subnet with at least two free IPs, and zone-b had none.
  7. A cleanup script deleted both replicas of one API at once, the second self-inflicted problem, because they ran on two different drifted nodes and both replacements went Pending.
  8. Prometheus was OOMKilled against a 1536Mi memory limit and entered a crash loop.

The article also lists eight read-only checks, with the specific values that indicate each problem, for auditing a similar cluster before starting.

Why it matters

Many EKS clusters now sit exactly where this one did: past standard support, on a Karpenter release whose supported range ends below the cluster version, and on AL2 nodes with no upgrade path at all. The alarming log line is informational; the real work is CRD version migration in Terraform, do-not-disrupt annotations, subnet IP capacity and image pull limits. The ordered sequence (Karpenter, then node OS, then control plane) and the timestamped failure modes turn a confusing compatibility error into a practical checklist.

  • #eks
  • #kubernetes
  • #karpenter
  • #aws
  • #cluster-upgrade

Related posts