LLM Inference Optimization and KV Caching Explained

Updated on Sep 29, 2026

9 min read

As chat assistants, coding tools, and retrieval-augmented generation (RAG) systems move from demos into production, the cost and latency of answering each request become important design constraints. LLM inference optimization is the set of techniques used to improve how quickly and efficiently a trained language model serves requests. A key technique is key-value (KV) caching, which saves attention data from earlier tokens so a model does not calculate it again at every generation step. This explainer covers how the cache works, what it costs, and when it helps.

What Is LLM Inference Optimization and KV Caching?

Inference is the process of running a trained model to produce an output. A text-generating Transformer usually processes an input prompt and then predicts output tokens one at a time. At every step, self-attention relates the current token to previous tokens in the sequence.

For each Transformer layer, attention derives key and value vectors from the tokens it has processed. These vectors represent information the model will consult as it generates the next token. A KV cache retains those vectors in accelerator memory. When a new token arrives, the model calculates and adds that token’s keys and values, then uses the accumulated cache instead of recalculating the old ones.

The Hugging Face Transformers KV-cache guide describes how generation caches hold key/value states and the memory trade-offs of different cache implementations. Caching is part of inference, not training: it does not change model weights or teach the model new information.

The Problem KV Caching Solves

Autoregressive generation is sequential: the model cannot produce the next token until it has selected the current one. Without a cache, each generation step can repeat key and value projections for the entire prefix, including tokens already handled. As the conversation grows, that repeated work wastes computation and increases the time between generated tokens.

A KV cache avoids that repeated projection work, but it introduces a different constraint: memory. Cached tensors grow with the sequence length and the number of active requests. For a decoder-only model with uniform attention layers, an approximate cache size for one sequence is:

KV bytes ≈ 2 × layers × KV heads × head dimension × tokens × bytes per value

The factor of two accounts for keys and values. For example, 32 layers with 8 KV heads, a head dimension of 128, 8,192 tokens, and two-byte values use about 1 GiB for one sequence, before allocator overhead. Actual architectures can differ, and a server with multiple active sequences needs space for each one. Long prompts, long outputs, and concurrent users can therefore fill GPU memory even when the model weights fit.

Inference systems balance several measures rather than optimizing a single number. Time to first token (TTFT) includes prompt processing and queueing; inter-token latency measures the delay between generated tokens; throughput measures completed tokens or requests over time; and memory use limits how many sequences can run concurrently. A configuration that increases throughput with larger batches may raise queueing time or memory pressure.

How KV Caching Works During Inference

Generation has two broad phases. During prefill, the model processes the input prompt and calculates attention states for its tokens. During decode, it predicts output tokens one at a time. At each decode step, the model computes the new token’s query, key, and value. It appends the new key and value to the cache, then lets the query attend to the available keys and values.

This saves the repeated calculation of past keys and values; it does not make attention to the full history free. The attention operation still has to use the relevant cached context, and longer contexts can require more memory reads and computation. As a result, KV caching improves generation efficiency but cannot remove every cost that grows with context length.

Serving multiple users adds an allocation problem. Requests arrive and finish at different times, and their prompts have different lengths. A naive implementation that reserves a maximum-size contiguous cache for each request can leave GPU memory unused or fragmented. The PagedAttention paper describes managing KV data in blocks, inspired by virtual memory, so sequences can occupy non-contiguous blocks. The vLLM project applies this approach in an inference-serving engine.

Some serving systems can also reuse cached blocks when separate requests share the same prompt prefix, such as a system instruction. The tokens must match in the conditions the engine uses for cache identity; similar wording alone does not guarantee a hit. See the vLLM prefix-caching documentation for its block reuse behavior.

Components and Optimization Techniques

Technique	What it changes	Main benefit	Main trade-off
Per-request KV cache	Retains prior keys and values during one generation	Avoids recomputing old token projections	Uses memory proportional to context length
Prefix caching	Reuses matching prompt blocks across requests	Reduces repeated prefill work for shared prefixes	Requires prefix matches and cache residency
Paged KV allocation	Stores cache blocks independently rather than reserving one large contiguous region	Improves allocation flexibility and memory use under varied sequence lengths	Adds engine-managed block allocation and scheduling
Continuous batching	Admits new requests as others finish instead of waiting for a fixed batch	Increases accelerator utilization and aggregate throughput	Scheduling choices can affect request latency
Grouped-query or multi-query attention	Uses fewer KV heads than query heads in a model architecture	Reduces cache size and memory traffic	Depends on model architecture and may affect quality trade-offs
KV-cache quantization	Stores cache values at lower precision	Fits more context or active requests in memory	Requires supported kernels and validation for quality and speed
Optimized attention kernels	Changes how attention computation moves data and uses temporary memory	Can improve attention compute efficiency	Does not by itself eliminate the persistent KV cache

These techniques address different bottlenecks. Prefix reuse helps when prompts share long, stable beginnings; continuous batching helps keep hardware busy across variable request arrivals; and cache quantization targets memory capacity. FlashAttention-style kernels can reduce intermediate memory traffic for attention, but should not be confused with storing less persistent KV state. A deployment may combine several techniques, so measure each change against the same workload.

Real-World Use Cases

Conversational assistants retain the earlier turns of a conversation while generating the next response. Cache size grows with retained history, so applications need a context limit or a deliberate history-summarization policy.
RAG services often send the same instruction and formatting rules with different retrieved passages. Prefix caching may reuse the shared beginning, while newly retrieved text still has to be processed. The cache does not replace retrieval, update source documents, or make a stale answer current. For retrieval systems that combine semantic matches with exact terms, see hybrid search for RAG.
Code completion and agents can repeatedly evaluate requests with common scaffolding or tool instructions. Reuse can shorten repeated prompt processing, but the changing code and tool results still require fresh model work.
High-concurrency APIs use batching and paged allocation to serve requests of different lengths. The relevant target is useful throughput within an acceptable latency and memory budget, not maximum batch size by itself.
Tracing production requests can show how model inference fits alongside retrieval, retries, and tool calls. See OpenTelemetry for LLM Production Observability for an instrumentation walkthrough.

Getting Started: Enable KV Caching with vLLM

vLLM is an open-source serving engine for language models. Run it in an environment supported by its GPU installation guide; available hardware, drivers, and platform support vary, so check that guide rather than assuming the same setup works on every operating system. On a supported environment with uv installed, install vLLM and start an OpenAI-compatible server:

uv pip install vllm --torch-backend=auto

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --enable-prefix-caching \
  --max-model-len 4096

The model is downloaded when the server first starts. The maximum model length caps the combined prompt and generated sequence; choose a value that the model supports and the available GPU memory can accommodate. The cache for a request’s own tokens is part of generation; the prefix-caching flag enables reuse where request prefixes match.

Check that the server responds and send a short request:

curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","messages":[{"role":"user","content":"Explain KV caching in one sentence."}],"max_tokens":64}'

For a useful benchmark, repeat representative requests with and without prefix reuse, including requests that do and do not share a prefix. Track TTFT, inter-token latency, throughput, peak GPU memory, cache hit behavior, and output quality. Hold the model, prompt lengths, concurrency, and hardware constant while comparing configurations. If the process runs out of memory, first reduce concurrency or maximum context, then evaluate smaller cache precision or a model with fewer KV heads; each option has different quality and latency consequences.

Common Misconceptions

“A KV cache makes generation constant-time.” It avoids recomputing prior keys and values, but attention still reads cached history, and each output token still requires model computation.
“Prefix caching helps any two similar prompts.” Reuse depends on matching token prefixes and the serving engine’s cache rules. Different wording, tokenization, or request state can prevent a hit.
“KV caching shrinks the model.” It stores temporary per-request state; it does not reduce the model’s parameter count. It can increase total memory use even as it reduces repeated computation.
“A cache hit guarantees lower end-to-end latency.” Queueing, prefill, network time, batch scheduling, and output length also contribute. Measure the complete request path under realistic traffic.

Transformer Architecture Deep Dive explains the attention mechanism that produces keys and values.
Vector Databases for RAG covers how systems retrieve context before a language model generates an answer.
Prompt Engineering for LLMs explores how instructions and context shape model requests.

Changelog

Initial publication.

Last updated: September 29, 2026