Kubernetes zero downtime deployment before canary tooling

A kubernetes zero downtime deployment depends on three independent contracts: Ready, Registered, Drained. The application can serve traffic; the load balancer recognises its target; the retiring process finishes accepted work. These are not three interchangeable settings.

First, separate ALB-generated errors from application-generated errors. In ALB access logs, compare elb_status_code with target_status_code: a missing target response is not an application HTTP error. Correlate CloudWatch’s HTTPCode_ELB_5XX_Count and HTTPCode_Target_5XX_Count with timestamped application logs. AWS’s ALB troubleshooting guidance distinguishes these failure sources. A 502 alone does not establish a shutdown race.

For EKS rolling update 502 errors, use this decision tree to choose an investigation, not declare a cause:

• Errors as new Pods start → Ready. Inspect application readiness. Does the readiness probe pass before dependencies and request handlers actually work?
• Kubernetes Ready before ALB target Healthy → Registered. Inspect target registration, health checks and pod readiness gates. Kubernetes readiness alone cannot establish ALB health.
• Errors around old-Pod deletion, or afterwards → Drained. Inspect Kubernetes graceful shutdown and connection lifetime. Compare termination signals, the preStop hook, process exit and ALB connection draining.

Capture Pod transitions, EndpointSlice conditions and application events against the same timeline:

kubectl get pods -n app -l app=api -w
kubectl get endpointslices -n app -l kubernetes.io/service-name=api -o yaml
kubectl logs -n app POD --timestamps

The 21 October practitioner experiment motivates testing fresh and persistent connections separately: they produced different outcomes using kind and iptables. Its proposed conntrack explanation is not established for production EKS, IP targets or other dataplanes. Its failure counts are not EKS statistics.

Rollout completion reports controller state, not an end-to-end client acceptance test. Diagnosing the failing contract before adding canary tooling is part of Cloudline’s Kubernetes and deployment reliability engineering work.

A running container has not earned traffic. For a Kubernetes zero downtime deployment, separate three decisions:

• startupProbe: startup protection. Readiness and liveness checks wait until this succeeds. Slow initialisation should not trigger premature restarts.
• readinessProbe: traffic eligibility. Fail it when this instance cannot serve the relevant request. Failure removes readiness; it does not restart the container.
• livenessProbe: restart detection. Fail it for a stuck process that restarting could repair, not an unavailable shared database.

Application contract: implement /startupz, /readyz and /livez on port 8080. Return HTTP 200 for success and 503 for failure. Readiness must check the serving path, not merely whether the HTTP listener responds.

If every useful request requires PostgreSQL, readiness may need to reflect that dependency. If cached responses remain useful, keep those instances eligible. Never make a shared-dependency incident restart the fleet through liveness checks.

Replace YOUR_REGISTRY/api:YOUR_IMMUTABLE_TAG with your application's immutable image reference. Create the app namespace first if it does not exist.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: app
spec:
  replicas: 3
  minReadySeconds: 10
  progressDeadlineSeconds: 600
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      terminationGracePeriodSeconds: 75
      containers:
        - name: api
          image: YOUR_REGISTRY/api:YOUR_IMMUTABLE_TAG
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
          startupProbe:
            httpGet:
              path: /startupz
              port: 8080
            periodSeconds: 2
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 2
            failureThreshold: 1
          livenessProbe:
            httpGet:
              path: /livez
              port: 8080
            periodSeconds: 10

These are example budgets, not measured recommendations: roughly 60 seconds for startup, two-second readiness checks and 75 seconds for termination. Tune them against observed startup and request durations. Section four explains Kubernetes graceful shutdown and the purpose and timing of a preStop hook; none is included here.

maxUnavailable: 0 needs schedulable surge capacity and genuinely serving replicas. This extra Pod requests 500m CPU and 512Mi memory. See Understanding Kubernetes Pod Basics for resource requests and scheduling background.

minReadySeconds: 10 requires ten continuous ready seconds before Deployment availability. It is a stability interval, not confirmation from AWS. EKS rolling update 502 errors can persist after these probes pass; ALB registration and pod readiness gates are separate checks.

A passing readiness probe does not mean the ALB has registered the Pod. This gap causes EKS rolling update 502 errors: Kubernetes removes old capacity before AWS can use its replacement.

IP targets:
client → public ALB → Pod IP:application port

Instance targets:
client → public ALB → node:NodePort → cluster Service dataplane → Pod
                                      (potentially on another node)

Identify your actual Service dataplane: kube-proxy’s configured mode, or a replacement such as Cilium eBPF. Do not assume every cluster uses iptables.

kubectl get configmap kube-proxy -n kube-system -o yaml
kubectl get daemonsets -n kube-system

AWS Load Balancer Controller Pod readiness gates apply to IP targets, not instance targets. They hold Pod readiness until the corresponding ALB target becomes healthy.

1. Create and label the namespace.
2. Create the matching Service and IP-target ALB Ingress.
3. Wait for its IP TargetGroupBinding, with the correct Service reference.
4. Only then create replacement Pods.

kubectl create namespace app --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace app elbv2.k8s.aws/pod-readiness-gate-inject=enabled --overwrite

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: app
spec:
  selector:
    app: api
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api
  namespace: app
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /readyz
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80
YAML

kubectl get targetgroupbindings -n app -o yaml

This example assumes app=api, port 8080 and a working /readyz endpoint. Confirm spec.targetType: ip and spec.serviceRef before proceeding. Injection happens at Pod creation; labelling the namespace does not retrofit existing Pods.

kubectl rollout restart deployment/api -n app
kubectl get pods -n app -l app=api -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.readinessGates}{"\n"}{end}'
kubectl describe pod -n app POD

aws elbv2 describe-target-groups --target-group-arns "$TG_ARN"
aws elbv2 describe-target-health --target-group-arn "$TG_ARN"

Use the TargetGroupBinding’s ARN for TG_ARN. Correlate Pod IPs and gate conditions with target states and reason codes:

• Initial: registration or initial health checks; inspect Elb.RegistrationInProgress or Elb.InitialHealthChecking.
• Healthy: ALB health checks pass; confirm the Pod gate also becomes true.
• Draining: deregistration underway, usually Target.DeregistrationInProgress; this is not replacement readiness.

If registration stalls, inspect controller logs, health-check paths and security-group reachability to the target port.

kubectl logs -n kube-system deployment/aws-load-balancer-controller --since=10m
kubectl get endpointslices -n app -l kubernetes.io/service-name=api -o yaml

For instance targets, healthy nodes cannot prove individual Pod registration. Inspect EndpointSlices and node-to-Pod routing instead. Changing target type is an ingress migration, not a harmless live annotation tweak.

Reassessing ingress ownership? ECS Gateway vs ALB: which ingress should you own? covers that decision; its ECS-specific settings do not apply to EKS.

Pod deletion starts two concurrent paths: endpoint and ALB updates propagate while the kubelet starts local shutdown. There is no guaranteed “traffic stops, then SIGTERM” sequence. Pod readiness gates protect startup; they do not prove shutdown is safe.

The grace-period countdown includes the preStop hook, which runs before the normal stop signal. If propagation needs breathing room, add this under the Deployment’s Pod template:

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 75
      containers:
        - name: app
          lifecycle: {preStop: {exec: {command: ["/bin/sh", "-c", "sleep 10"]}}}

This requires /bin/sh and sleep in the image. Ten seconds is an illustrative propagation allowance, not a universal fix. The process must remain able to serve throughout that allowance.

Budget the 75-second baseline explicitly: 10 seconds propagation + up to 60 seconds application drain, including cleanup + 5 seconds margin. Longer requests need a different budget.

Let active requests finish

This Node.js HTTP/1.1 example stops accepting connections on SIGTERM, finishes requests, then closes PostgreSQL resources. Its 60-second force-exit deadline is shorter than the approximately 65 seconds remaining:

import http from 'node:http';
import pg from 'pg';

const pool = new pg.Pool();
let draining = false;

const server = http.createServer(async (req, res) => {
  if (req.url === '/ready') {
    res.writeHead(draining ? 503 : 200).end();
    return;
  }
  if (draining) res.setHeader('Connection', 'close');
  try {
    await pool.query('SELECT 1');
    res.end('OK\n');
  } catch {
    res.writeHead(503).end('Database unavailable\n');
  }
});
server.listen(3000);

process.once('SIGTERM', () => {
  draining = true;
  const deadline = setTimeout(() => process.exit(1), 60_000);
  server.close(async (error) => {
    try {
      if (error) throw error;
      await pool.end();
      clearTimeout(deadline);
      process.exitCode = 0;
    } catch {
      process.exit(1);
    }
  });
});

Current Node.js versions, since v19, also close idle keep-alive connections through server.close(). Do not start with server.closeAllConnections(): it also closes active requests. Require exec-form container startup so Node receives SIGTERM:

CMD ["node", "server.mjs"]

Align ALB draining with process lifetime

Client-to-ALB and ALB-to-target connections use separate pools; they are not one socket. Inspect the target server’s keep-alive timeout against the ALB idle timeout. A target closing earlier can cause EKS rolling update 502 errors despite otherwise sound Kubernetes graceful shutdown.

Configure ALB connection draining through the controller-owned Ingress:

metadata:
  annotations:
    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=60

This bounds draining after deregistration begins. It must fit actual request durations and process lifetime; the two countdowns do not necessarily start together. Avoid out-of-band AWS mutations of controller-managed attributes.

WebSockets, HTTP/2 and streaming need protocol-specific drain and reconnect handling, not this HTTP/1.1 example alone.

A green rollout is not proof of a kubernetes zero downtime deployment. Run this experiment from a CI runner outside the cluster, against an authorised EKS-backed public hostname. Each connection mode runs ten VUs; neither retries failed requests.

Use a read-only, authenticated application route, not a health check. The example expects JSON containing an exact marker, plus version and pod identifiers. The route must perform application work, disable server-side caching and support a bounded delay_ms parameter.

• Fresh: disables connection reuse, exposing connection-establishment failures.
• Keepalive: retains client connections. This does not pin requests to an ALB backend socket: ALB manages backend connections independently.
• Slow requests: every tenth request exercises work lasting less than the configured drain budget.

Save this as deployment-test.js. Supply latency limits from the service SLO, including a separate limit for deliberately slow requests. Request IDs correlate client failures with application and ALB timelines.

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Counter } from 'k6/metrics';

const mode = __ENV.MODE;
const expectedStatus = Number(__ENV.EXPECTED_STATUS || 200);
const failures = {
  resets: new Counter('connection_resets'),
  timeouts: new Counter('request_timeouts'),
  '502': new Counter('responses_502'),
  '503': new Counter('responses_503'),
  '504': new Counter('responses_504'),
};

export const options = {
  vus: 10,
  duration: __ENV.DURATION,
  noConnectionReuse: __ENV.MODE === 'fresh',
  maxRedirects: 0,
  tags: { mode },
  thresholds: {
    http_req_failed: ['rate==0'],
    checks: ['rate==1'],
    'http_req_duration{case:normal}': [
      `p(95)<${__ENV.NORMAL_P95_MS}`,
    ],
    'http_req_duration{case:slow}': [
      `p(95)<${__ENV.SLOW_P95_MS}`,
    ],
  },
};

http.setResponseCallback(http.expectedStatuses(expectedStatus));

export default function () {
  const started = Date.now();
  const id = `${started}-${mode}-${__VU}-${__ITER}`;
  const slow = __ITER % 10 === 9;
  const kind = slow ? 'slow' : 'normal';
  const url = `${__ENV.BASE_URL.replace(/\/$/, '')}${__ENV.TEST_PATH}`
    + `?request_id=${id}&delay_ms=${slow ? __ENV.SLOW_MS : 0}`;

  const response = http.get(url, {
    timeout: `${__ENV.REQUEST_TIMEOUT_SECONDS}s`,
    headers: {
      Authorization: `Bearer ${__ENV.API_TOKEN}`,
      'Cache-Control': 'no-cache, no-store',
      'X-Request-ID': id,
    },
    tags: { name: 'deployment-probe', case: kind },
  });

  let body = {};
  try { body = response.json(); } catch (_) {}

  const passed = check(response, {
    'exact expected status': r => r.status === expectedStatus,
    'exact response marker': () => body.marker === __ENV.EXPECTED_MARKER,
    'version and Pod present': () =>
      Boolean(body.version && body.pod),
  });

  const error = response.error || '';
  failures.resets.add(/reset/i.test(error) ? 1 : 0);
  failures.timeouts.add(
    response.error_code === 1050 || /timeout|timed out/i.test(error) ? 1 : 0
  );
  for (const status of ['502', '503', '504']) {
    failures[status].add(response.status === Number(status) ? 1 : 0);
  }

  console.log(JSON.stringify({
    event: 'request', id, mode, case: kind,
    started_at: new Date(started).toISOString(),
    finished_at: new Date().toISOString(),
    status: response.status, passed,
    latency_ms: response.timings.duration,
    version: body.version || null, pod: body.pod || null,
    error, error_code: response.error_code,
  }));
  sleep(0.2);
}

The wrapper warms both clients before changing the image. Its duration covers warm-up, baseline, the ten-minute rollout timeout and a post-rollout window. Set CONNECTION_TIMER_SECONDS to the largest relevant configured connection timer; the post-window must exceed that and ALB connection draining.

#!/usr/bin/env bash
# Save as deployment-test.sh; run with Bash, kubectl, AWS CLI and k6 installed.
set -euo pipefail

: "${PUBLIC_URL:?}" "${IMAGE:?}" "${API_TOKEN:?}" "${TEST_PATH:?}"
: "${EXPECTED_MARKER:?}" "${NORMAL_P95_MS:?}" "${SLOW_P95_MS:?}"
: "${DRAIN_SECONDS:?}" "${CONNECTION_TIMER_SECONDS:?}"
: "${SLOW_MS:?}" "${REQUEST_TIMEOUT_SECONDS:?}"
: "${TARGET_GROUP_ARN:?}" "${ALB_ARN:?}"
: "${APP_SELECTOR:?}" # For example: app=api

BASELINE_SECONDS=${BASELINE_SECONDS:-30}
WARMUP_SECONDS=30
ROLLOUT_SECONDS=600
largest=$(( DRAIN_SECONDS > CONNECTION_TIMER_SECONDS \
  ? DRAIN_SECONDS : CONNECTION_TIMER_SECONDS ))
POST_SECONDS=${POST_SECONDS:-$((largest + 60))}

(( POST_SECONDS > largest ))
(( SLOW_MS > 0 && SLOW_MS < DRAIN_SECONDS * 1000 ))
(( REQUEST_TIMEOUT_SECONDS * 1000 > SLOW_MS ))
TOTAL_SECONDS=$((WARMUP_SECONDS + BASELINE_SECONDS \
  + ROLLOUT_SECONDS + POST_SECONDS + 60))

export API_TOKEN TEST_PATH EXPECTED_MARKER NORMAL_P95_MS SLOW_P95_MS
export SLOW_MS REQUEST_TIMEOUT_SECONDS
export EXPECTED_STATUS="${EXPECTED_STATUS:-200}"
export DURATION="${TOTAL_SECONDS}s"

mkdir -p artefacts
cd artefacts
cp ../deployment-test.js .

stamp() { date -u +%Y-%m-%dT%H:%M:%SZ; }

# Record Kubernetes, kubelet, container runtime and installed dataplane images.
kubectl version -o yaml > kubernetes-version.yaml
kubectl get nodes -o json > nodes-and-runtime.json
kubectl get deployment,daemonset -A -o yaml > controller-cni-dataplane.yaml
kubectl get deployment/api -n app -o yaml > deployment-before.yaml
kubectl get targetgroupbindings -A -o yaml > target-bindings.yaml
k6 version > k6-version.txt
aws elbv2 describe-target-groups \
  --target-group-arns "$TARGET_GROUP_ARN" > target-type.json
aws elbv2 describe-target-group-attributes \
  --target-group-arn "$TARGET_GROUP_ARN" > drain-attributes.json
aws elbv2 describe-load-balancer-attributes \
  --load-balancer-arn "$ALB_ARN" > alb-attributes.json

# Poll throughout the test, including after rollout status returns.
(
  while :; do
    stamp
    kubectl get pods -n app -l "$APP_SELECTOR" -o wide
    kubectl get events -n app --sort-by=.metadata.creationTimestamp
    aws elbv2 describe-target-health \
      --target-group-arn "$TARGET_GROUP_ARN"
    kubectl logs -n app -l "$APP_SELECTOR" --all-containers=true \
      --prefix --timestamps --since=20s --max-log-requests=20
    sleep 10
  done
) > cluster-alb-application-timeline.log 2>&1 &
observer=$!
fresh='' keepalive=''
cleanup() {
  kill "$observer" ${fresh:+"$fresh"} ${keepalive:+"$keepalive"} \
    2>/dev/null || true
}
trap cleanup EXIT

MODE=fresh BASE_URL="$PUBLIC_URL" k6 run --out json=fresh.json deployment-test.js \
  > fresh.log 2>&1 &
fresh=$!
MODE=keepalive BASE_URL="$PUBLIC_URL" k6 run --out json=keepalive.json deployment-test.js \
  > keepalive.log 2>&1 &
keepalive=$!

# Require successful application requests from both clients before baseline.
warmed=0
for ((i=0; i<WARMUP_SECONDS; i++)); do
  if grep -q '"passed":true' fresh.log \
    && grep -q '"passed":true' keepalive.log; then
    warmed=1
    break
  fi
  sleep 1
done

set +e
if (( warmed )); then
  sleep "$BASELINE_SECONDS"
  stamp > rollout-start.txt
  kubectl set image -n app deployment/api api="$IMAGE" \
    > set-image.log 2>&1
  image_rc=$?
  kubectl rollout status -n app deployment/api --timeout=10m \
    > rollout-status.log 2>&1
  rollout_rc=$?
else
  image_rc=1
  rollout_rc=1
  echo "Both clients did not warm successfully" > rollout-status.log
fi
stamp > rollout-status-finished.txt

# Never terminate load merely because Kubernetes reports success.
wait "$fresh"; fresh_rc=$?
wait "$keepalive"; keepalive_rc=$?
set -e
printf 'set_image=%s\nrollout=%s\nfresh=%s\nkeepalive=%s\n' \
  "$image_rc" "$rollout_rc" "$fresh_rc" "$keepalive_rc" > exit-statuses.txt
kubectl get deployment/api -n app -o yaml > deployment-after.yaml

(( image_rc == 0 && rollout_rc == 0 \
   && fresh_rc == 0 && keepalive_rc == 0 ))

Archive artefacts even when CI fails. Export central application logs and ALB access logs for the same UTC window; allow for ALB log-delivery delay. Polling alone can miss a short-lived Pod. Record controller, CNI, target type, runtime and dataplane versions alongside results.

Report resets, timeouts, 502s, 503s and 504s separately, preserving original failures without retries. Calculate the latest failure offset from rollout-start.txt; retain failures after rollout completion too. Compare baseline and fixed configurations without claiming success from Pod readiness gates or a preStop hook alone.

Keep this evidence in your CI/CD and monitoring workflows, rather than treating a successful rollout command as the release gate.

A Kubernetes zero downtime deployment needs overlap capacity, not just correct lifecycle settings. Terminating Pods can consume resources beyond replicas + maxSurge.

Consider an illustrative 12-replica service with maxSurge: 3, requesting 1 vCPU and 2 GiB per Pod. A late-rollout snapshot could contain 12 replacements plus 11 terminating old Pods: 23 Pods, not 15. That is 11 extra vCPU and 22 GiB above steady state—not a guaranteed peak.

Integrating an illustrative rollout timeline gives:

Non-terminating surplus: 540 Pod-seconds
Terminating Pods:     1,440 Pod-seconds
Total extra:          1,980 Pod-seconds
Requested capacity:   0.55 vCPU-hours + 1.10 GiB-hours

This allocation is not an EC2 bill. Existing node headroom could make incremental EC2 compute zero. Otherwise, bin-packing, allocatable resources, AZ placement, IP availability and node lifetime determine spend.

Assume four additional workload slots per node: 11 extra Pods need three nodes. At a hypothetical £0.20/node-hour:

• 15-minute billed lifetime: 3 × £0.20 × 0.25 = £0.15/deployment; £45/month for 300 deployments.
• Ten-minute lifetime: £0.10/deployment; £30/month at the same frequency.

These are modelling inputs, not quoted London AWS prices. They exclude EBS, ALB, data transfer and VAT. See Understanding Cloud Cost Dynamics for broader planning.

ALB Service-selector cutover is not inherently atomic; Argo documents a blue/green ALB downtime risk. Neither canary nor blue/green fixes broken draining. Require backwards-compatible schema changes whichever strategy you choose.

Start with rolling when lifecycle and compatibility tests pass. Base shutdown waits on measured ALB connection draining, not arbitrary sleeps. Add controllers only for an exposure or cutover requirement—not to hide lifecycle failures.

For a deployment reliability review, Talk to a cloud engineer.

Start with the failed request, not a new deployment controller. Establish whether the break happens during readiness, target registration or shutdown, then prove the fix with both connection modes through the public endpoint. Budget for terminating Pods as well as surge capacity. If your team needs help connecting those timelines, Cloudline Consulting can review the deployment path, identify the smallest practical remediation and define a repeatable acceptance test before you invest in more platform tooling.