Kubernetes Kueue: Workload Admission and Quotas

Kubernetes Kueue is a workload queuing system for clusters that need to decide which batch or AI jobs may start before their Pods compete for placement. Platform teams use it to manage shared capacity across namespaces, teams, and accelerators; workload owners submit work to a queue and can see whether it is waiting or admitted. This explainer covers Kueue’s admission model, quota sharing, preemption, and a small setup you can test on a Kubernetes cluster.

What Is Kubernetes Kueue?

Kueue is a Kubernetes-native job queueing and admission controller. It evaluates workloads against quota and policy configured by cluster administrators. If a workload fits the available quota, Kueue admits it and allows its workload controller to create Pods; otherwise, the workload remains queued. Kueue complements the Kubernetes scheduler rather than replacing it: admission governs when a workload may consume a reserved share, while the scheduler chooses nodes for its Pods.

The Kueue project is part of the Kubernetes SIG Scheduling ecosystem. Its overview documentation describes the workload admission flow and integrations. Kueue handles Kubernetes Jobs and can integrate with other supported job APIs, including JobSet. It is most useful when multiple workloads contend for limited resources and operators want a consistent queue and quota policy instead of relying on every team to launch work at once.

Kueue does not run the job’s computation, add cluster nodes, or guarantee that an admitted Pod will immediately become Ready. The underlying controller still creates the workload’s Pods, and the Kubernetes scheduler still places them. For the built-in behavior of a Kubernetes Job itself, see the Job controller documentation.

Why Kueue Exists

The default scheduler handles individual Pods that are ready to run. That works well for services and independent tasks, but it is not a complete capacity policy for a shared batch cluster. If dozens of training or data-processing jobs arrive together, they can all begin competing for the same CPU, memory, and accelerator capacity. Some may start partially, occupy scarce resources, or wait indefinitely behind other work without a clear team-level admission policy.

A queue provides an explicit point to decide whether a whole workload should start. Administrators can define nominal quotas and permitted resource flavors, place queues into a cohort for resource sharing, and use priority or preemption policies to manage contention. Workloads that cannot be admitted can wait without launching their Pods. This makes resource allocation more legible, though it does not remove the need for capacity planning or sensible resource requests.

Kueue is especially relevant for bursty and accelerator-intensive workloads. It can govern when a Job or a grouped workload such as a JobSet gets to start, while other components continue to handle the workload’s lifecycle and hardware. Kubernetes JobSet describes grouping related Jobs; Kueue addresses the separate question of admission and quota.

How Kueue Works

An owner creates a supported workload and associates it with a namespaced LocalQueue. The queue maps to a cluster-scoped ClusterQueue, which defines quota and policy. Kueue represents the request as a Workload, checks whether it can be admitted, and records the decision in Kubernetes status. On admission, the integration updates the underlying job resource so its Pods can be created. The ordinary Kubernetes scheduler then attempts to place those Pods.

Admission is a resource-accounting decision, not a physical reservation of a particular machine. Kueue’s ResourceFlavor can associate a quota flavor with node labels and tolerations, so the workload’s Pods can target a compatible pool. But cluster capacity may change, storage or affinity rules may still block placement, and an accelerator driver must still make usable devices available. Kubernetes Dynamic Resource Allocation for GPUs explains a separate API for device allocation.

Quota sharing and fairness

A ClusterQueue defines how much of a resource is nominally available for a flavor. For example, a queue might receive a baseline of four CPUs and eight GiB of memory. If it belongs to a cohort with other queues, configured borrowing and lending rules can let one queue temporarily use quota that another queue is not using. Cohorts provide a sharing boundary; they do not mean every queue has unlimited access to every cluster resource.

Fair sharing is used to help distribute scarce quota among competing workloads according to their usage and the configured policy. It complements nominal quotas and borrowing: a cohort makes resources shareable, while fair sharing influences which queue should receive the next available admission opportunity. Operators should inspect the Kueue version’s policy options and documentation because queue ordering, preemption, and sharing behavior depend on configuration.

Components and Key Concepts

• Workload: Kueue’s representation of work under admission control. An integration translates a supported resource, such as a Job, into the resource requests Kueue evaluates. The Workload status and events help explain whether it is pending, inadmissible, or admitted.
• LocalQueue: A namespaced entry point for workload owners. It refers to a ClusterQueue, allowing different namespaces to expose their own queue names while sharing or separating cluster policy.
• ClusterQueue: The cluster-scoped quota and admission policy. It specifies which namespaces may use it and how resource quotas are grouped by flavor.
• ResourceFlavor: A named description of a resource pool, often associated with node labels or tolerations. For GPUs, flavors can distinguish pools, but device discovery and allocation remain the responsibility of the relevant Kubernetes driver and APIs.
• Cohort: A group of ClusterQueues that can share unused quota when borrowing and lending are configured. The cohort gives queues a common sharing boundary without erasing their separate nominal allocations.
• Admission checks and integrations: Integrations connect Kueue to job APIs, while admission checks can coordinate external readiness or capacity conditions where configured. The exact supported workload types and checks depend on the installed components.
• Preemption: A policy-driven way to reclaim quota when a higher-priority or otherwise eligible workload cannot be admitted. Depending on the policy and integration, Kueue may suspend or otherwise preempt workloads; this is not the same as arbitrary eviction of any running Pod.

The ClusterQueue concepts guide describes quota, flavors, cohorts, and preemption in more detail. Kueue operates through Kubernetes APIs and controllers, so the relevant objects and their status are inspectable with standard cluster tools.

Real-World Use Cases

Shared machine-learning infrastructure can provide teams with queues for training and evaluation. A team submits a workload to a queue, while the platform team controls the queue’s accelerator or CPU quota. For deployments that need to allocate particular GPU devices to Pods, Kueue admission can be combined with Dynamic Resource Allocation; admitting a workload and allocating a device solve related but distinct problems.

Batch data processing often arrives in waves and does not need every job to begin immediately. Kueue can hold work until its resource requests fit the configured quota, then admit it as capacity becomes available. This helps prevent unbounded concurrency from swamping shared infrastructure, though operators still need job-level timeouts, retries, and failure handling.

Multi-team clusters can map namespace-local queue names to cluster queues and apply different quotas or namespace selectors. Cohorts can allow teams to borrow unused capacity while retaining nominal shares. Administrators can then make resource policies explicit and observe which workloads are waiting for admission.

Distributed jobs can use Kueue alongside a workload API such as JobSet. JobSet describes related Jobs, their lifecycle, and networking; Kueue determines whether the grouped workload may start under the cluster’s queue policy. For the JobSet model and its controller, see Kubernetes JobSet explained.

Getting Started with Kueue

Use a disposable or non-production Kubernetes cluster for this example. It installs the Kueue controller and CRDs, creates a simple CPU-and-memory quota, then submits a suspended Job that Kueue can admit. The official installation guide documents supported installation methods and version requirements. Pinning the release makes the example reproducible; check the project documentation before selecting a version for your cluster.

Install Kueue and wait for its controller to become ready:

VERSION=v0.20.0
kubectl apply --server-side -f "https://github.com/kubernetes-sigs/kueue/releases/download/${VERSION}/manifests.yaml"
kubectl wait --for=condition=Available deployment/kueue-controller-manager -n kueue-system --timeout=120s
kubectl get pods -n kueue-system

The next manifest creates a default flavor, a ClusterQueue with nominal quota, and a LocalQueue in the default namespace. An empty namespace selector allows namespaces to use this ClusterQueue; in a multi-tenant cluster, replace it with an explicit selector and apply suitable access controls.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: default
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: team-a
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["cpu", "memory"]
      flavors:
        - name: default
          resources:
            - name: cpu
              nominalQuota: 4
            - name: memory
              nominalQuota: 8Gi
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: user-queue
  namespace: default
spec:
  clusterQueue: team-a

Apply the configuration and verify that the API objects exist:

kubectl apply -f queues.yaml
kubectl get resourceflavors
kubectl get clusterqueues
kubectl get localqueues -n default

Submit a suspended Job that requests less than the quota. The queue label associates it with the LocalQueue; Kueue’s Job integration evaluates the workload and controls when it is unsuspended.

apiVersion: batch/v1
kind: Job
metadata:
  name: queued-demo
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  suspend: true
  parallelism: 2
  completions: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: busybox:1.36.1
          command: ["sh", "-c", "echo admitted; sleep 30"]
          resources:
            requests:
              cpu: 1
              memory: 1Gi

Save the job as queued-demo.yaml, apply it, and inspect the workload and Job state:

kubectl apply -f queued-demo.yaml
kubectl get workloads -n default
kubectl describe workload -n default
kubectl get jobs,pods -n default
kubectl describe clusterqueue team-a

The workload status and events show whether it has been admitted or is waiting and can point to a queue, quota, or flavor mismatch. If it does not progress, check that the LocalQueue references an active ClusterQueue, the namespace is allowed by its selector, the Job has the queue label, and the requested resources are covered by the ClusterQueue’s resource groups. Once admitted, inspect Pod events separately: a Kueue admission does not prove that nodes exist or that the scheduler can satisfy every Pod constraint. If you are finished, delete the example Job and queue resources.

Common Misconceptions

“Kueue is another Kubernetes scheduler.” Kueue admits workloads against queue policy; the Kubernetes scheduler still binds each created Pod to a node.

“Admitted means running.” Admission means the workload passed Kueue’s checks and can proceed. Image pulls, node capacity, storage, device setup, and Pod scheduling can still delay or prevent execution.

“A quota is a hard physical reservation.” Quota is part of admission accounting. It does not pin a machine or guarantee immediate access to infrastructure, and cohort borrowing is limited by configuration and actual capacity.

“Kueue replaces a workflow or distributed-job controller.” It decides whether supported work can start. Job controllers still manage completion and retries, while JobSet or another workload API expresses group structure and lifecycle behavior.

Related Articles

• Kubernetes JobSet explained covers grouped Jobs and distributed-workload lifecycle management.
• Kubernetes Dynamic Resource Allocation for GPUs explains device claims and accelerator allocation.
• Kubernetes cluster autoscaling and node provisioning explains how unschedulable Pods can lead to additional node capacity.
• Kubernetes architecture explained introduces the scheduler and control-plane components involved after admission.

Changelog

• 2026-10-12: Initial publication; installation and admission concepts checked against the Kueue project documentation.