Kueue Job Pending? Prove You Need More GPUs Before Buying

A Kueue Job pending does not prove GPU exhaustion. Use our four-gate capacity test: queue entitlement → external admission checks → topology fit → Pod placement. This is an investigation model, not four strictly sequential controllers.

Trace the path: Job → namespaced LocalQueue → ClusterQueue policy and Kueue ResourceFlavor selection. AdmissionCheck controllers attach to admission. An admitted Job, once resumed, produces Pods for Kubernetes scheduling. Topology-Aware Scheduling (TAS) can constrain admission as well as later placement.

You need Bash, kubectl, jq and cluster read access. Set your namespace and Job name:

NS=ml
JOB=training
JOB_UID=$(kubectl -n "$NS" get job "$JOB" -o jsonpath='{.metadata.uid}')
kubectl -n "$NS" get workloads.kueue.x-k8s.io -l "kueue.x-k8s.io/job-uid=$JOB_UID" -o json | jq '.items[] | {name:.metadata.name,queue:.spec.queueName,conditions:.status.conditions,checks:.status.admissionChecks}'

This UID-based Workload investigation route is documented by Kueue. Zero matches? Check the Job’s kueue.x-k8s.io/queue-name label, Job integration, controller health, namespace and read permissions before assuming quota exhaustion. An access error is not an empty result.

Select WL only when exactly one match exists; otherwise stop and investigate:

WL=$(kubectl -n "$NS" get workloads.kueue.x-k8s.io \
  -l "kueue.x-k8s.io/job-uid=$JOB_UID" -o json |
  jq -er 'if (.items | length) == 1 then .items[0].metadata.name
          else error("Expected exactly one Workload") end') || exit 1

kubectl -n "$NS" get workload "$WL" -o json | jq -r '.status.conditions[]? | [.type,.status,.reason,.message] | @tsv'
kubectl -n "$NS" get job "$JOB" -o json | jq '{suspend:.spec.suspend,conditions:.status.conditions}'

Inspect Job suspension separately: suspension alone does not identify the blocked gate. Route by evidence:

• No quota reservation: section two investigates entitlement and insufficient quota.
• Quota reserved, but not admitted: section three investigates admission checks.
• Topology-specific message: section four investigates where the requested GPUs fit.
• Admitted, with stalled Pods: section five follows the scheduler; inspect Pod resource requests, not just GPU utilisation.

Missing conditions prove none of these failures. Read the reasons and messages before choosing a branch.

Use a disposable lab with Kueue serving kueue.x-k8s.io/v1beta2, matching the official quick start. The documented installation reference is v0.20.4; this is not an instruction to upgrade production. Check available resources and discover controller images read-only:

kubectl api-resources --api-group=kueue.x-k8s.io
kubectl get deployments -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,IMAGES:.spec.template.spec.containers[*].image'

This fixture requests 100m CPU against zero CPU quota. Its dedicated ClusterQueue has no Cohort, so borrowing cannot conceal the block. No GPU hardware is needed.

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Namespace
metadata:
  name: kueue-proof
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: proof-default
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: proof-cq
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: kueue-proof
  resourceGroups:
  - coveredResources: [cpu, memory]
    flavors:
    - name: proof-default
      resources:
      - name: cpu
        nominalQuota: 0
      - name: memory
        nominalQuota: 128Mi
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: proof
  namespace: kueue-proof
spec:
  clusterQueue: proof-cq
---
apiVersion: batch/v1
kind: Job
metadata:
  name: quota-proof
  namespace: kueue-proof
  labels:
    kueue.x-k8s.io/queue-name: proof
spec:
  suspend: true
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: pause
        image: busybox:1.36
        command: [sh, -c, 'sleep 60']
        resources:
          requests:
            cpu: 100m
            memory: 32Mi
YAML

Follow the previous section’s UID route, not a guessed Workload name. These commands require jq; allow the controller time to create the Workload.

JOB_UID=$(kubectl -n kueue-proof get job quota-proof -o jsonpath='{.metadata.uid}')
WL=
for attempt in $(seq 1 60); do
  WL=$(kubectl -n kueue-proof get workloads -o json |
    jq -r --arg uid "$JOB_UID" \
    '.items[] | select(any(.metadata.ownerReferences[]?; .uid == $uid)) | .metadata.name')
  [ -n "$WL" ] && break
  sleep 2
done
test -n "$WL" || exit 1

kubectl -n kueue-proof wait "workload/$WL" \
  --for=condition=QuotaReserved=False --timeout=120s

proof_status() {
  kubectl -n kueue-proof get job quota-proof \
    -o jsonpath='suspend={.spec.suspend}{"\n"}'
  kubectl -n kueue-proof get workload "$WL" -o json |
    jq -r '.status.conditions[]? |
      select(.type == "QuotaReserved" or .type == "Admitted") |
      "\(.type)=\(.status) message=\(.message)"'
  kubectl -n kueue-proof get pods -o json |
    jq -r --arg uid "$JOB_UID" \
    '[.items[] | select(any(.metadata.ownerReferences[]?; .uid == $uid))] |
      "Job Pods=\(length)"'
}
proof_status

Expected before-output, normalised — not captured execution:

suspend=true
QuotaReserved=False
Condition message: insufficient cpu quota for flavor proof-default
Job Pods=0

Repair only the CPU entitlement:

kubectl patch clusterqueue proof-cq --type=json -p='[{"op":"replace","path":"/spec/resourceGroups/0/flavors/0/resources/0/nominalQuota","value":1}]'

The equivalent corrected YAML fragment retains the resource group and memory entry. Do not assume arrays merge by name.

spec:
  resourceGroups:
  - coveredResources: [cpu, memory]
    flavors:
    - name: proof-default
      resources:
      - name: cpu
        nominalQuota: 1
      - name: memory
        nominalQuota: 128Mi

kubectl -n kueue-proof wait "workload/$WL" \
  --for=condition=Admitted=True --timeout=120s
proof_status
kubectl -n kueue-proof wait job/quota-proof \
  --for=condition=Complete --timeout=180s

Expected after-output, normalised: QuotaReserved=True, Admitted=True, suspend=false, then completion on a schedulable lab node with image access. These are executable recovery assertions, not a test report.

For GPU queues, resolve the same accounting path:

CQ=$(kubectl -n kueue-proof get localqueue proof \
  -o jsonpath='{.spec.clusterQueue}')
kubectl get clusterqueue "$CQ" -o json |
  jq '{resourceGroups: .spec.resourceGroups,
       flavorsReservation: .status.flavorsReservation,
       cohort: .spec.cohort}'

Compare aggregate PodSet GPU requests with the selected flavor’s quota, existing reservations, borrowing limits and Cohort policy. nominalQuota is an entitlement, not installed hardware. This CPU lab proves admission accounting, not GPU placement; CLI-based Pod sizing explains why requests matter more than utilisation graphs.

kubectl delete namespace kueue-proof
kubectl delete clusterqueue proof-cq
kubectl delete resourceflavor proof-default

QuotaReserved=True and Admitted not True: quota is reserved, but admission is incomplete. For this Kueue job pending case, inspect admission checks before requesting more GPUs. All required checks must return Ready.

kubectl -n "$NS" get workload "$WL" -o json | jq '.status.admissionChecks // []'
kubectl get clusterqueue "$CQ" -o json | jq '{conditions:.status.conditions,checks:.spec.admissionChecks,strategy:.spec.admissionChecksStrategy}'
kubectl get admissioncheck "$CHECK" -o yaml

Workload check states describe this admission attempt. They are not the AdmissionCheck object’s Active condition:

• Pending: awaiting the controller’s decision.
• Ready: this check permits admission.
• Retry: this attempt failed; requeueing is required.
• Rejected: the controller rejected the Workload.

Active=True means the AdmissionCheck is operational, not that your Workload passed. A missing or inactive referenced check can make the ClusterQueue inactive. That fault does not necessarily produce the quota-reserved evidence above.

Read spec.controllerName to identify the responsible controller; it is not necessarily a Deployment name. Find the implementation managing that identifier. Follow Workload and related-object events, inspect spec.parameters references, then read that controller’s logs.

For a provisioning controller, distinguish broken reconciliation or invalid parameters from an actual capacity failure. A missing parameter object is configuration trouble. A generated ProvisioningRequest reporting unavailable capacity is a different incident: inspect its conditions and provider errors before buying GPUs.

Correct the reference, not the verdict

In this dedicated example, provisioning-check already exists and is Active. Correct a mistyped ClusterQueue reference with:

spec: {admissionChecks: [provisioning-check]}

This applies only to the simple admissionChecks form. Preserve an existing flavour-scoped admissionChecksStrategy.

Confirm recovery: check state Ready, Workload Admitted=True, then Job resumed. Never patch a Workload check’s status to Ready, remove a required production check, or unsuspend the Job to force progress.

Inspect the selected or candidate Kueue ResourceFlavor, not the queue’s marketing name. “GPU capacity” says nothing about accelerator compatibility or placement.

kubectl get resourceflavor "$FLAVOR" -o yaml
kubectl get nodes -o json | jq '.items[] | {name:.metadata.name,labels:.metadata.labels,taints:.spec.taints,gpu:.status.allocatable["nvidia.com/gpu"]}'

The flavour’s nodeLabels constrain placement. Its nodeTaints describe taints used in flavour selection; tolerations are separately configured and automatically added to admitted Pods. Check actual node taints too: tolerating a node does not select it.

Allocatable is not unallocated. Inspect requests from bound Pods and node descriptions, including Ready and Unschedulable state. Use Pod request inspection, not GPU utilisation, to assess scheduler headroom.

kubectl get pods -A --field-selector spec.nodeName="$NODE" -o yaml
kubectl describe node "$NODE"

Count GPUs inside the required topology

For Topology-Aware Scheduling (TAS), inspect the flavour’s .spec.topologyName, its Topology object, PodTemplate annotations and Workload assignments:

TOPOLOGY=$(kubectl get resourceflavor "$FLAVOR" -o jsonpath='{.spec.topologyName}')
kubectl get topology "$TOPOLOGY" -o yaml
kubectl get workload "$WORKLOAD" -n "$NAMESPACE" -o jsonpath='{.status.admission.podSetAssignments}'

Hypothetically, eight workers request one GPU each. Eight compatible GPUs exist, split four-and-four across two racks, with no suitable single-rack alternative. Requiring all eight workers within one rack cannot fit. TAS can prevent admission; this is not merely a post-admission scheduler problem.

Required placement, on the worker PodTemplate:

metadata:
  annotations:
    kueue.x-k8s.io/podset-required-topology: topology.example.com/rack

Alternatively, remove the required annotation and use preferred placement:

metadata:
  annotations:
    kueue.x-k8s.io/podset-preferred-topology: topology.example.com/rack

The annotation value must name a real configured topology level. Relax placement only after the training owner accepts bandwidth and completion-time consequences. For hard requirements, add capacity inside an eligible domain or correct demonstrably wrong labels. Never relabel nodes to invent physical locality.

Admitted=True hands responsibility to Pod creation and scheduling. For “kueue job pending”, this is the next diagnostic boundary—not proof that GPUs are missing.

kubectl -n "$NS" get pods -l "batch.kubernetes.io/job-name=$JOB" -o wide
kubectl -n "$NS" describe job "$JOB"
POD=$(kubectl -n "$NS" get pods \
  -l "batch.kubernetes.io/job-name=$JOB" \
  -o jsonpath='{.items[0].metadata.name}')
# Run these once a Pod exists:
kubectl -n "$NS" describe pod "$POD"
kubectl -n "$NS" get pod "$POD" -o yaml

Inspect .spec.schedulingGates, .spec.schedulerName, PodScheduled conditions and container waiting reasons.

• No Pods: investigate Job-controller creation failures through Job events.
• SchedulingGated: identify the gate owner; do not remove policy gates blindly.
• FailedScheduling: follow the event’s constraint.
• Gang scheduler installed: inspect its permit/plugin logs and PodGroup status.
• PodScheduled=True, image pull waiting: placement succeeded; troubleshoot image delivery.

Prove admission is not placement

After correcting quota, use a fresh fixture copy. First establish that no node matches:

kubectl get nodes -l cloudline.example.com/fixture=absent
# Expected: No resources found

# yq v4; fixture.yaml is the original source manifest.
yq '.metadata.name = "placement-proof" |
  .spec.template.spec.nodeSelector = {"cloudline.example.com/fixture": "absent"}' \
  fixture.yaml > placement-proof.yaml
kubectl -n "$NS" create -f placement-proof.yaml
JOB=placement-proof

Without TAS, the empty ResourceFlavor allows admission, but nodeSelector is a hard placement constraint. Repeat the diagnostics above. Normalised expected results:

Admitted=True
suspend=false
PodScheduled=False
FailedScheduling: ... didn't match Pod's node affinity/selector

Repair only this disposable Job. Do not assume an admitted Job’s PodTemplate is patchable.

kubectl -n "$NS" delete job placement-proof --wait=true
yq 'del(.spec.template.spec.nodeSelector)' placement-proof.yaml |
  kubectl -n "$NS" create -f -
kubectl -n "$NS" wait pod \
  -l batch.kubernetes.io/job-name=placement-proof \
  --for=condition=PodScheduled --timeout=120s
kubectl -n "$NS" wait job/placement-proof \
  --for=condition=Complete --timeout=300s

On EKS, inspect relevant Karpenter NodePool and EC2NodeClass constraints. Use Spot/On-Demand fallback guidance for evidence-led provisioning changes, not every Pending Pod.

Clean up while Kueue remains running; these resources must be dedicated to the lab:

kubectl -n "$NS" delete jobs --all --wait=true
kubectl delete namespace "$NS" --wait=true
kubectl delete clusterqueue "$CQ" --wait=true
kubectl delete resourceflavor "$FLAVOR" --wait=true

A Kueue Job pending is not a purchase order. Close the four-gate test with evidence:

Provisioning shortages can surface through a check before admission: Admitted=True is not a purchasing prerequisite. Do not increase quota for admission-controller failures, physical placement constraints or intentional tenant boundaries. Never remove the queue label or force suspension off to bypass Kueue.

Measure the wait you actually mean

Track pending counts and p95 admission wait using Kueue’s metrics:

sum by (cluster_queue,status) (kueue_pending_workloads)

histogram_quantile(0.95, sum by (le,cluster_queue) (rate(kueue_admission_wait_time_seconds_bucket[30m])))

Add kueue_admission_checks_wait_time_seconds to isolate reservation-to-admission delay. Admission histograms observe successful transitions, not the continuing age of never-admitted jobs. Creation-to-now submission age and requeue-aware admission wait are different measurements.

This returns the oldest currently unadmitted submission age in seconds, excluding finished and inactive Workloads; zero means none:

kubectl get workloads.kueue.x-k8s.io -A -o json |
jq '
  now as $now |
  [.items[]
   | select(.spec.active != false)
   | select(any(.status.conditions[]?;
       .type == "Finished" and .status == "True") | not)
   | select(any(.status.conditions[]?;
       .type == "Admitted" and .status == "True") | not)
   | ($now - (.metadata.creationTimestamp | fromdateiso8601))]
  | max // 0
'

Compare admission-to-PodsReady separately. A ten-minute objective is a team-selected SLO, not an industry benchmark.

Illustrative cost scenario — not a provider quote

Eight additional GPUs × 24 hours × an assumed £3/GPU-hour = 192 GPU-hours and £576 incremental compute spend. Assume they remain provisioned and billable while the same queue configuration blocks admission. Buying them changes nothing about that blocker.

For an eight-GPU instance, the per-GPU rate allocates the whole-instance charge. This excludes VAT, storage, networking, control-plane charges and management fees. It is not £576 of proven savings or lost revenue.

AWS announced EKS Auto Mode GPU management-fee reductions in July 2026, effective 1 July: 35% for G-series; 60% for P-series and Trainium. These are management-fee reductions, not equivalent EC2 compute-price cuts.

For a capacity-versus-policy diagnosis, request a batch-platform bottleneck assessment. Bring one Job manifest, its Workload conditions, relevant queue configuration, scheduler events and a queue-wait sample. Expect a diagnosis, not a guaranteed percentage saving.

A pending training job needs a demonstrated cause, not an automatic hardware order. Trace its Workload, identify the blocked gate and change only the configuration or capacity that the evidence implicates. Then verify admission, Pod readiness and useful training progress before declaring recovery. If the evidence crosses too many ownership boundaries, Cloudline Consulting can help with a batch-platform bottleneck assessment. Bring one delayed Job and its queue history; leave with a prioritised plan for fixing policy, placement or genuine capacity constraints.