Kubernetes cgroup v2 migration go or no-go

Use Cloudline’s six evidence gates as your Kubernetes upgrade checklist. Each needs an artefact and a named individual accountable for it, not just a team. An unknown blocks approval.

• Policy: applicability record; platform lead owns it.
• Inventory: node and runtime register; infrastructure lead owns it.
• Compatibility: application and cgroup v2 memory monitoring results; application lead owns it.
• Image: reproducible image manifest; image maintainer owns it.
• Rehearsal: canary and rollback evidence; SRE lead owns it.
• Release: signed decision and exception register; release lead owns it.

For a Kubernetes cgroup v2 migration, distinguish upstream defaults from provider behaviour. A control-plane version alone does not establish node compatibility.

Temporary exception, not a migration: this KubeletConfiguration fragment permits cgroup v1 where applicable.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false

The 6 October 2026 blog describes removal as scheduled for v1.38. However, KEP-5573 specifies no fixed removal timeline: no earlier than v1.38, not a guaranteed deadline.

Do not declare all managed compute unsupported from an upstream default. EKS AL2023 migration needs joined-up Kubernetes, OS and runtime planning—the scope of Cloudline’s cloud, DevOps and Kubernetes services.

Inventory every Linux node across explicitly selected contexts. For a Kubernetes cgroup v2 migration, an image name is not evidence. Custom AMIs and replacement nodes can differ from the pool they supposedly belong to.

Save this as audit.sh, then run bash audit.sh production-uk staging-uk > nodes.tsv. It requires kubectl and jq, never changes your active context, and creates no Pods.

#!/usr/bin/env bash
set -euo pipefail
(( $# )) || { echo 'Usage: audit.sh CONTEXT [CONTEXT...]' >&2; exit 2; }
for cmd in kubectl jq; do
  command -v "$cmd" >/dev/null || {
    echo "Missing dependency: $cmd" >&2; exit 2;
  }
done
printf 'context\tnode\tpool\tos\tkernel\truntime\tkubelet\tcgroup\n'
for ctx in "$@"; do
  nodes=$(kubectl --context="$ctx" --request-timeout=20s get nodes -o json)
  while IFS= read -r node; do
    name=$(jq -r '.metadata.name' <<<"$node")
    metrics=''
    if ! metrics=$(kubectl --context="$ctx" --request-timeout=10s \
      get --raw "/api/v1/nodes/$name/proxy/metrics"); then
      printf 'WARNING: %s/%s metrics unavailable\n' "$ctx" "$name" >&2
      metrics=''
    fi
    cg=$(awk '/^kubelet_cgroup_version([{ ]|$)/ {print $NF; exit}' <<<"$metrics")
    case "$cg" in
      1|2) ;;
      *) cg=UNKNOWN
         printf 'WARNING: %s/%s cgroup UNKNOWN\n' "$ctx" "$name" >&2 ;;
    esac
    jq -r --arg ctx "$ctx" --arg cg "$cg" '[
      $ctx, .metadata.name,
      (.metadata.labels["eks.amazonaws.com/nodegroup"] //
       .metadata.labels["karpenter.sh/nodepool"] // "UNLABELLED"),
      .status.nodeInfo.osImage, .status.nodeInfo.kernelVersion,
      .status.nodeInfo.containerRuntimeVersion,
      .status.nodeInfo.kubeletVersion, $cg
    ] | @tsv' <<<"$node"
  done < <(jq -c '.items[] | select(.status.nodeInfo.operatingSystem == "linux")' <<<"$nodes")
done

Use an approved audit identity. Access to nodes/proxy is sensitive: it exposes kubelet endpoints, not merely harmless inventory. Do not grant it casually to make this script pass.

KEP-5573 identifies kubelet_cgroup_version for migration monitoring. Values 1 and 2 provide runtime evidence; denied access, unreachable kubelets and missing metrics remain UNKNOWN. Unknown does not mean compatible.

Reconcile what Kubernetes cannot see

• Match registered nodes to EC2 instances, including stopped instances and machines that failed to join.
• Compare managed node-group and Auto Scaling group desired capacity with registered capacity. Investigate every mismatch.
• Inspect scale-from-zero node groups and Karpenter NodeClasses. Record their resolved AMIs and launch-template versions, not just today’s running images.

A node where the kubelet fails to start may never enter this report. Check bootstrap logs and effective failCgroupV1 settings rather than treating an empty pool as clean.

For node-level confirmation, use an approved host session, such as Systems Manager Session Manager:

stat -fc %T /sys/fs/cgroup/
findmnt -R /sys/fs/cgroup
cat /proc/1/cgroup

Run these against the host, not an arbitrary application container. cgroup2fs confirms the unified mount; tmpfs needs investigation for legacy or hybrid layouts.

Collect effective kubelet configuration, the runtime’s cgroup driver, AMI ID and launch-template version separately. OS labels alone prove neither configuration nor compatibility. Cloudline’s Understanding Cloud Risks explains the operational cost of undocumented infrastructure.

Make application compatibility a release gate for your Kubernetes cgroup v2 migration. Record deployed versions against this matrix; version checks alone cannot earn a pass.

Kubernetes lists compatibility floors including OpenJDK 8u372 and 11.0.16, standalone cAdvisor 0.43.0 and automaxprocs 1.5.1. These are not recommended lifecycle targets. Its guidance also warns that unsupported Node.js versions can detect host memory instead of container limits.

AWS documents older JDK8 builds sizing heaps from node memory on AL2023, causing OOM exceptions and Pod restarts. Run inside a representative container; compare detected memory with its configured limit.

kubectl exec memory-probe -c app -- java -XshowSettings:system -version
kubectl get pod memory-probe -o jsonpath='{.spec.containers[?(@.name=="app")].resources}'

For cgroup v2 memory monitoring, retrieve that container’s runtime ID from the API:

kubectl get pod memory-probe -o jsonpath='{.status.containerStatuses[?(@.name=="app")].containerID}'

On its node, set CID to that ID without containerd://. Follow the upstream PID-to-path method:

PID=$(sudo crictl inspect "$CID" | jq -r '.info.pid')
CG=$(sudo awk -F: '$1=="0"{print $3}' /proc/$PID/cgroup)
sudo grep . "/sys/fs/cgroup$CG"/memory.{current,max,stat,events}

Compare old/new nodes under equal load: cgroup counters, Prometheus working-set/RSS series and CloudWatch ingestion. Working set includes kernel and recently accessed memory, not just application heap. Induce a memory-limit breach in an isolated workload; verify OOM counters, restarts, missing-series alerts and actual alert delivery.

Do not silence changed graphs by raising limits: establish whether accounting or application behaviour changed first. Use Cloudline’s Kubernetes Pod sizing guide for requests, limits and measured headroom.

Keep the EKS AL2023 migration separate from the Kubernetes version change. Record actual current and target minors in the Kubernetes upgrade checklist; “upgrade EKS” is not a sequence.

1. For a planned 1.35 → 1.36 upgrade, qualify AL2023 with 1.35 nodes under the unchanged 1.35 control plane, where supported.
2. Complete the OS replacement and drain the old pools.
3. Upgrade the control plane to 1.36, then the nodes. Never run a kubelet newer than the control plane.

Check EKS prerequisites and available AL2023 AMIs for those exact minors. An April 2024 migration example is not a current compatibility matrix.

AL2023 uses nodeadm and NodeConfig, not copied AL2 bootstrap.sh arguments. Within the launch template’s MIME user data, include an application/node.eks.aws part. Validate field support against the selected kubelet before using this fragment:

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  kubelet:
    config:
      cgroupDriver: systemd
      failCgroupV1: true

Custom-AMI and self-managed paths also need cluster name, API endpoint, certificate authority and service CIDR under spec.cluster. With supported failCgroupV1 enforcement, the kubelet fails to start on cgroup v1. AL2023 cgroup v1 is not a supported migration destination.

The documented VPC CNI minimum is 1.16.2; select an add-on release compatible with both migration-stage and target minors. Test IMDSv2-dependent credentials, Pod Identity or IRSA, DNS, EBS mounts and log shipping.

Standard managed node groups without a specified custom AMI require blue/green replacement. Custom-AMI launch templates have different update mechanics; prefer a separate canary group anyway.

The diagram shows an unchanged control plane above old and canary pools, shared networking and storage, and identical observability paths. Cloudline’s platform engineering services cover reproducible Terraform and image pipelines.

Budget overlap, not just replacement. For this Kubernetes cgroup v2 migration planning example—not an AWS quote—a 12-node fleet adds three equivalent canary nodes, one per Availability Zone.

The assumed effective rate is £0.20 per node-hour. Generalise with Σ(extra nodes × hours × effective hourly rate). Add EBS, monitoring ingestion, network traffic and applicable tax separately. See Understanding Cloud Cost Dynamics for effective rates and temporary-capacity trade-offs.

Three nodes are not a universal minimum. Check allocatable CPU, memory and Pod slots after DaemonSet overhead, plus subnet IPs and AWS quotas.

1. Retain the old pool. Create canaries labelled migration=cgroup2 and tainted migration=cgroup2:NoSchedule.
2. Select representative workloads explicitly using node selectors and tolerations. Ensure every required DaemonSet tolerates the taint.
3. Run load, batch, storage and restart tests. Then migrate one old node at a time.

kubectl get pdb -A
kubectl drain "$OLD_NODE" --ignore-daemonsets --timeout=15m

Drain uses eviction and respects PDBs; --disable-eviction bypasses that protection. An emptyDir warning requires a deliberate data-loss decision, not an automatic extra flag.

Rehearse with three service replicas, a PDB specifying minAvailable: 2, and topology spread across zones using maxSkew: 1. Deliberately make one replica unready, then drain a node hosting another: eviction should block. Restore readiness, correct replica placement or revise the budget—not force-delete Pods. PDBs do not prevent involuntary failures.

Example team-set thresholds: one complete peak/batch cycle; all required agents healthy; an alert delivered within five minutes; no unexplained OOMs; p95 latency within the agreed SLO.

Rehearse rollback: restore workload placement to retained old nodes, drain the canaries and confirm recovery. Retain sufficient old capacity; constrain autoscaler and Karpenter changes so they cannot remove your escape route.

A Kubernetes cgroup v2 migration is GO only when all six gates pass. A booting node is not a release decision.

If the kubelet fails to start, investigate the cause. A failCgroupV1 override cannot turn failed compatibility, monitoring or recovery tests green.

Postpone the Kubernetes upgrade if safe operation depends on an untested OS migration. That is defensible only while the current release remains within an acceptable security and provider-support window. Give every HOLD a named owner, remediation date, explicit expiry and support-cost review. Expiry triggers reassessment, not automatic approval or indefinite delay.

If the current platform is already exposed, agree a separately tested emergency path. Do not compress fleet discovery and an EKS AL2023 migration into the upgrade window.

Keep the two rollback decisions separate

Node-pool rollback restores the previous node image and capacity. Control-plane rollback is a separate operation: current EKS documentation allows eligible rollback to the previous minor within seven days, subject to prerequisites and compatibility checks. It is not an unconditional downgrade and does not replace the OS return-path test.

Bring this evidence to your cluster upgrade-readiness assessment:

• Fleet inventory, AMI provenance and reproducible build records.
• Workload, agent, heap-sizing and memory-monitoring results.
• Drain and return-path logs; subnet headroom; approved overlap budget.
• Exception owners, expiry dates and support-cost decisions.

For a second pair of eyes on your Kubernetes upgrade checklist, talk to a cloud engineer.

A Kubernetes cgroup v2 migration is ready when the evidence agrees: the fleet is accounted for, the replacement image is reproducible, applications recognise their limits, monitoring remains useful and workloads survive both the move and the return journey. If an essential gate is still unknown, separate the OS work from the version upgrade rather than gambling on both. Cloudline Consulting can help turn those unknowns into a practical plan. Talk to a cloud engineer about a cluster upgrade-readiness assessment.