Kubernetes Gateway API Inference Extension Explained

Updated on Oct 11, 2026

9 min read

Running a language model on Kubernetes is only part of serving it: a platform must also decide which model-server replica should handle each request. The Kubernetes Gateway API Inference Extension adds inference-aware routing to Gateway API so platform teams can direct traffic to pools of model servers using request and endpoint signals. This guide explains where the extension fits, how its resources cooperate, and what to check before exposing an inference endpoint.

What Is the Kubernetes Gateway API Inference Extension?

The Kubernetes Gateway API Inference Extension is a set of APIs and components for routing requests to self-hosted generative AI models on Kubernetes. It builds on Kubernetes Gateway API, whose controllers configure gateways and route HTTP traffic. The extension adds an inference-specific backend, InferencePool, and an optional Endpoint Picker (EPP) integration for selecting a model-server replica.

The extension does not run a model or replace its serving engine. Model servers still process prompts and generate tokens; Kubernetes schedules their pods; and a compatible gateway handles network traffic. The extension connects those pieces so routing can consider information about inference workloads rather than treating every replica as interchangeable.

The official extension documentation describes its API resources, implementation roles, and request flow. The project repository publishes the API, reference components, and conformance tests. Check the gateway implementation’s documentation as well: supporting Gateway API alone does not guarantee that it supports this inference extension.

The Problem It Solves

A basic HTTP load balancer can distribute requests among healthy backends, but LLM replicas do not always have equivalent capacity for every request. One replica may have a warm prefix cache or a requested adapter already loaded; another may have a longer queue or less accelerator memory available. A round-robin choice can send work to a replica that is healthy but poorly positioned to serve it.

The consequences include wasted prefill work, uneven queue times, poor accelerator utilization, and unpredictable latency. These effects become more visible when requests differ in prompt length, output size, or model requirements. The serving engine’s own scheduler decides which sequences run inside a replica; a gateway-level inference router addresses a different question: which replica should receive the request?

How Inference-Aware Routing Works

A request passes through several cooperating layers:

client
  -> Gateway API Gateway and listener
  -> HTTPRoute selects an InferencePool backend
  -> Endpoint Picker evaluates a request and pool endpoints
  -> selected model-server pod receives the request
  -> model server returns the response through the gateway

An HTTPRoute attaches to a Gateway and routes matching traffic to a backend. For inference traffic, that backend can be an InferencePool rather than an ordinary Kubernetes Service. The pool identifies the model-serving pods and their target ports. Where an EPP is configured, the gateway’s extension integration consults it to select an endpoint from that pool.

An EPP can use the information available from the model-serving implementation to choose a replica. Examples include queue length, KV-cache use, and available LoRA adapters. Which signals are exposed, how they are collected, and how they influence a decision depend on the model server, EPP, and gateway integration; the API does not guarantee that every implementation uses every signal. The HTTP protocol’s methods, fields, and status codes remain governed by the HTTP Semantics standard, RFC 9110; inference-aware endpoint selection is an additional routing decision above that protocol contract.

Component	Responsibility	What it does not do
Gateway and controller	Accept traffic and implement Gateway API routes	Run or schedule the model
`HTTPRoute`	Match incoming requests and select an inference backend	Pick the best pod within a pool by itself
`InferencePool`	Identify serving pods, target ports, and optional picker integration	Deploy model weights or guarantee that pods are ready
Endpoint Picker	Select an endpoint using supported request and endpoint information	Generate tokens or replace the model server’s internal scheduler
Model server	Load a model and process inference requests	Configure the cluster’s public gateway

This division lets teams keep the network entry point, route policy, serving pool, and per-replica selection concerns separate. It also means that a working HTTPRoute is not proof that the model is loaded, that endpoint metrics are fresh, or that a particular routing optimization is active.

Key Resources and Components

Gateway API resources remain the outer routing layer. A GatewayClass identifies the controller; a Gateway defines listeners; and HTTPRoute attaches HTTP rules to those listeners. Standard host, path, and header matching can route requests to an inference pool just as it can route traffic to other backends.

InferencePool is a namespaced custom resource. Its selector identifies the pods that serve a pool, while targetPorts specifies where the gateway should send traffic on those pods. Pools usually group replicas with compatible model and compute characteristics. An HTTPRoute can refer to one or more pools as backends, which supports route-level separation between serving groups.

Endpoint Picker is an extension component, not a required synonym for a model server. When configured, endpointPickerRef identifies its service and port. The API makes this reference optional, but a particular gateway integration may require it or have its own supported mode. The API’s failureMode controls behavior when the picker does not respond; FailClose is the API default, while FailOpen allows a gateway-selected endpoint as a fallback. Confirm the gateway’s behavior and the operational safety of a fallback before relying on it.

The project has changed component boundaries over time. In the v1.6 release line, the full-featured picker and body-based routing components moved to the llm-d project; the Gateway API Inference Extension repository continues to maintain the InferencePool API, conformance tests, and a lightweight reference picker. Teams should choose components from the implementation they actually deploy rather than assuming every example component is packaged in one repository.

Real-World Uses

Shared inference platforms can expose one Gateway while application teams route to model pools they are authorized to use.
Replicas with different cache or adapter state can use an EPP that considers supported endpoint signals when making a choice.
Model variants and capacity tiers can be represented by separate pools, with HTTPRoute rules directing requests to the appropriate backend.
Multi-cluster serving can use related project APIs and controllers to represent imported pools, where the chosen implementation supports them.

Inference-aware routing is most useful when there are several eligible replicas and meaningful differences between them. It does not fix a model that is overloaded everywhere, missing its weights, or unable to meet a latency objective. Capacity planning, admission control, and the model-serving engine’s scheduling remain necessary. For the per-replica scheduling layer, see LLM serving schedulers and continuous batching; for cache behavior inside inference engines, see LLM inference optimization and KV caching.

Getting Started: Route to an InferencePool

Start with a Kubernetes cluster, a Gateway API controller that supports the inference extension, the required CRDs, serving pods, and an Endpoint Picker if your implementation requires one. Install compatible versions using that gateway’s instructions before applying resources. Do not assume that installing the InferencePool CRD alone configures a gateway or deploys an EPP.

This example assumes an inference namespace with model-server pods labeled app: qwen-server, listening on port 8000, an EPP Service named qwen-epp on port 9002, and a Gateway named ai-gateway. Adjust the names, labels, ports, Gateway, and host to match the installed implementation.

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: qwen-pool
  namespace: inference
spec:
  selector:
    matchLabels:
      app: qwen-server
  targetPorts:
    - number: 8000
  endpointPickerRef:
    name: qwen-epp
    port:
      number: 9002
    failureMode: FailOpen
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: qwen-inference
  namespace: inference
spec:
  parentRefs:
    - name: ai-gateway
  hostnames:
    - models.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/
      backendRefs:
        - group: inference.networking.k8s.io
          kind: InferencePool
          name: qwen-pool

Save the resources as inference-route.yaml and apply them only after the named Gateway, serving pods, and EPP are available:

kubectl apply -f inference-route.yaml
kubectl get gateway ai-gateway -n inference
kubectl get httproute,inferencepool -n inference
kubectl describe httproute qwen-inference -n inference
kubectl describe inferencepool qwen-pool -n inference
kubectl get pods -n inference -l app=qwen-server

Check route and pool status conditions, resolved references, pod readiness, and controller or EPP logs. A rejected route can indicate unsupported backend kinds or an attachment problem; an empty pool can point to labels or namespace mistakes; and a reachable route that returns model errors may indicate a serving protocol or model issue. After confirming the Gateway address and hostname, send a small request using the model server’s documented API and inspect gateway, picker, and server metrics together. Test picker failure behavior in a non-production environment before using FailOpen for real traffic.

For a deeper explanation of the general Gateway, listener, and route model, see Kubernetes Gateway API explained. The inference extension adds a specialized backend and selection path to that model; it does not replace the underlying Gateway API controller.

Common Misconceptions

“The extension serves the model.” It coordinates routing to model-server pods. A serving runtime must still load and execute the model.
“Every Gateway API controller understands InferencePool.” InferencePool support is an additional implementation capability. Verify the exact controller and version, including any extension configuration it requires.
“The Endpoint Picker replaces the model server’s scheduler.” The picker selects a replica for a request; the engine scheduler manages execution of admitted requests within that replica.
“A successful route proves inference is healthy.” Route acceptance only covers part of the path. Pool membership, model readiness, picker health, capacity, and request compatibility also matter.

Changelog and Last Updated

2026-10-11: First publication.