NVIDIA Dynamo Architecture for Distributed LLM Serving
Serving a language model across many GPUs is not only a matter of starting more model replicas. The system must route each request to an available worker, account for reusable attention state, and respond when traffic or hardware capacity changes. NVIDIA Dynamo is an open-source orchestration framework for distributed generative-AI inference. It coordinates inference engines and worker pools; it does not replace the model or the engine that executes it.
Why NVIDIA Dynamo Is Being Discussed
Large models and mixed request workloads create several infrastructure problems at once. A long prompt can keep an accelerator busy with prefill while existing requests need decode steps. A new request may repeat a prompt prefix whose key-value (KV) state is already resident on another worker. A fixed number of replicas can also be too many during a quiet period and too few during a burst.
These are fleet-level coordination problems, not just model-kernel problems. Dynamo brings request routing, worker discovery, KV-state movement, and optional capacity planning into a serving stack. It is most relevant when operators need to coordinate multiple GPUs or nodes; a single model on one accelerator may be better served by its inference engine alone.
What Is NVIDIA Dynamo?
Dynamo is an orchestration layer that connects a client-facing request path to inference workers running engines such as vLLM, SGLang, or TensorRT-LLM. Its official documentation describes an open-source framework for serving generative AI, while the Dynamo project repository characterizes the project as a distributed inference serving framework.
The distinction between an engine and Dynamo matters. An engine executes model operations and manages its local scheduling and GPU memory. Dynamo can place that engine in a wider system, discover workers, and choose where requests or stages of a request should run. A deployment can keep prefill and decode together or use separate worker pools, depending on its backend and configuration.
The Problem Distributed Serving Solves
An inference server can manage batches on its own GPU, but a multi-worker service must make additional decisions. Which replica should receive a request? Does that replica have useful prompt state cached? Should prompt processing and token generation share the same GPU pool? What happens when a worker becomes unhealthy or a queue grows faster than capacity?
Sending every request to the least-busy worker may discard useful KV state and repeat prefill work. Always sending a request to the worker with the best cache match can overload that worker. Separating prefill and decode avoids some contention, but requires the KV state produced during prefill to be transferred or made available to a decode worker. That movement consumes bandwidth and may offset the benefit of separate pools.
Dynamo addresses these choices with routing and runtime components. It does not make inference work disappear: compute, memory, network capacity, and model compatibility still constrain the service. This is separate from the client protocol. RFC 9110 defines HTTP semantics at the request/response boundary; it does not specify how an inference system schedules GPU work or transfers KV state between workers.
How Dynamo Coordinates a Request
A useful mental model divides the system into a request path, a control path, and a state path. The exact components depend on the deployment, but the Dynamo architecture overview describes these as cooperating parts of the runtime.
| Part | Responsibility | Example decision or work |
|---|---|---|
| Frontend | Accepts and normalizes client requests | Receives an OpenAI-compatible chat request |
| Router | Selects eligible workers using load and, when available, KV information | Chooses a prefill worker, then a decode worker |
| Inference workers | Execute model computation through a supported backend | Process prompt tokens or generate output tokens |
| KV transfer and events | Move cache state and share cache lifecycle information | Make prefill state available to a decode worker |
| Control components | Discover and manage worker capacity | Adjust deployment resources when configured to do so |
In a disaggregated request, the frontend passes a request to the router. A prefill worker processes the prompt and creates KV state. The router then selects a decode worker, and the system transfers the required state so that worker can generate tokens. Dynamo documentation describes using NIXL for high-speed KV transfer between workers; the route and transfer details vary by backend. Generated tokens return through the request path to the client.
The router can consider both projected load and prefix-cache overlap. A worker with matching cache blocks may avoid repeating some prefill computation, but routing must still account for its current queue and capacity. Cache events help keep the router’s view useful as workers allocate, retain, or release state.
This design supports more than one topology. In an aggregated setup, the same worker handles prompt processing and generation. In a disaggregated setup, separate prefill and decode pools can be scaled or tuned independently. The Dynamo disaggregated-serving guide explains the KV handoff and worker-selection flow. The NVIDIA Dynamo Quickstart shows a local frontend-and-worker path before adding cluster orchestration.
Components and Key Concepts
Inference backends do the model execution. Dynamo integrates with engines including vLLM, SGLang, and TensorRT-LLM, but not every feature is available in every backend or release. Operators should check the current compatibility matrix before designing around a particular routing, cache, or disaggregation feature.
KV-aware routing uses cache overlap and worker load when selecting destinations. The goal is to reuse available prompt state without creating a hot spot. It is a scheduling input, not a guarantee of a cache hit: cache eviction, request differences, memory pressure, and worker availability can change the outcome.
KV transfer and storage solve related but distinct problems. Transfer makes state usable by another worker during a request handoff. Cache management can retain or offload reusable blocks across memory tiers. These mechanisms can reduce repeated work or extend effective cache capacity, but they add data movement and operational dependencies. LLM inference optimization and KV caching explains why cache capacity and reuse affect serving.
The control path manages desired capacity rather than generating tokens for each request. When configured, a Planner can use runtime signals and performance estimates to recommend or apply changes to prefill and decode worker counts. Kubernetes deployments can use an operator and custom resources to reconcile that desired state. These components are optional deployment choices, not prerequisites for understanding the basic request path.
A gateway is a separate boundary. A Dynamo frontend can be the entry point, or a Kubernetes inference gateway can apply platform-level routing and policy before forwarding to a worker. For the Kubernetes API and inference-aware gateway layer, see Kubernetes Gateway API Inference Extension.
Real-World Use Cases
- High-volume chat services can route requests across a worker fleet while measuring queueing, time to first token (TTFT), inter-token latency (ITL), and completed-request throughput.
- Long-context and retrieval services may benefit from KV-aware placement when prompts share prefixes. The gain depends on cache residency and the cost of moving state; a cache match alone does not ensure a faster response.
- Uneven prompt and generation workloads may justify separate prefill and decode pools. Teams can allocate different hardware or replica counts to each stage, provided the transfer path and extra GPUs are worth the coordination cost.
- Kubernetes inference platforms can combine model-serving workers with declarative deployment management, gateway policies, and autoscaling. Those surrounding platform features are separate from model execution and need their own operational monitoring.
The common requirement is enough scale or workload variation to justify a distributed runtime. For basic queueing, token budgets, and continuous batching within a single engine, see LLM serving schedulers and continuous batching. Speculative decoding and disaggregated LLM inference compares disaggregation with a different technique that proposes and verifies output tokens.
Getting Started: Run a Local Smoke Test
Start with the official Dynamo Quickstart and choose a backend and installation path supported by the available NVIDIA GPU and drivers. The quickstart’s container path starts a frontend and a backend worker; model access may require accepting the model’s license and providing a Hugging Face token. Use the worker launch command for the backend you selected rather than assuming every engine has the same options.
For the local frontend path, the documented command is:
python3 -m dynamo.frontend --discovery-backend file
Once the frontend and a worker are running, check readiness and send a small request. The model name must match the model served by the worker:
curl -sf http://localhost:8000/health && echo OK
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Explain Dynamo briefly."}],"max_tokens":50}'
This verifies the local request path, not distributed scaling or fault recovery. For a Kubernetes deployment, follow the version-matched deployment guide and example manifests in the Dynamo repository; do not apply a manifest without checking its API version, backend, and GPU requirements. Validate that the intended workers register and become ready before routing production traffic.
To evaluate whether distribution helps, compare an aggregated baseline with the disaggregated configuration using the same model, hardware class, and request mix. Include short and long prompts, output lengths, and concurrent arrivals. Measure TTFT, ITL, queue time, KV transfer volume or duration, GPU memory, worker utilization, errors, and cost per completed request. Increase one dimension at a time; a throughput gain that breaches latency targets or consumes more capacity may not be an improvement.
Common Misconceptions
- “Dynamo is an inference engine.” Dynamo coordinates a serving system; backend engines execute model computation.
- “Disaggregation always improves performance.” It can isolate prefill and decode, but it adds KV movement and can strand capacity in one pool.
- “KV-aware routing guarantees a cache hit.” Routing can use cache information, but state may be evicted, unavailable, or outweighed by load.
- “An OpenAI-compatible endpoint defines the internal system.” HTTP describes the client-facing exchange. Worker selection, backend scheduling, and KV transfer are implementation details behind that interface.
Related Articles
- LLM Serving Schedulers and Continuous Batching Explained
- LLM Inference Optimization and KV Caching Explained
- Speculative Decoding and Disaggregated LLM Inference
- Kubernetes Gateway API Inference Extension
Changelog and Last Updated
Last updated: November 16. Initial publication.

