NATS JetStream Durable Messaging Architecture

NATS JetStream durable messaging adds persistence and replay to NATS publish-subscribe messaging. It is useful to application developers and platform engineers who need services to communicate asynchronously without losing messages when a consumer is offline. This explainer follows a message from publication into a stream, through consumer acknowledgments and retention, and shows where JetStream differs from a partitioned event log such as Kafka.

Why NATS JetStream Is Being Discussed

Service architectures increasingly need both fast messaging and a way to recover work after a process or connection fails. A transient notification can be delivered through ordinary pub/sub, but an order or payment event may need to remain available until a worker can process it. JetStream makes that persistence part of the NATS server rather than requiring a separate broker product.

That does not make it a universal replacement for every queue or event-streaming system. Its subject-based routing, stream policies, and consumer model suit some workloads better than others. The important decision is whether the application needs short-lived work delivery, independently replayable records, strict ordering boundaries, or a combination of those behaviors.

What Is NATS JetStream?

NATS JetStream is the persistence and streaming layer built into NATS. A NATS server can receive messages published to subjects, store matching messages in named streams, and deliver them to consumers that keep server-side progress and acknowledgment state. The NATS project describes the broader system as lightweight messaging for cloud-native applications; JetStream adds durable storage to that messaging foundation.

The main entities are:

• A subject is a dot-separated message address, such as orders.created. Publishers send to a subject, and subscribers can use wildcard patterns to match related subjects.
• A stream stores messages whose subjects match its configured subject patterns. A stream has storage, replication, retention, and size or age limits.
• A consumer reads messages from a stream. It can be ephemeral or durable, push-based or pull-based, and configured with delivery and acknowledgment policies.

Core NATS and JetStream are related but have different delivery behavior. Core NATS is transient pub/sub: a subscriber that is not connected does not later receive missed messages. JetStream stores messages and makes them available for later delivery, subject to the stream’s retention policy.

The Problem Durable Messaging Solves

Direct service calls couple the sender’s success to the receiver’s availability. Transient pub/sub loosens that coupling, but a message can disappear if no subscriber is listening at delivery time. Retrying the sender alone may not help if it cannot tell whether the receiver completed work before a connection failed.

JetStream separates publication from processing. A publisher can receive a server acknowledgment for a stored message, while a consumer can read it later and acknowledge successful handling. If the consumer disconnects before acknowledging, the server can redeliver according to its configuration. This lets teams recover work after restarts and scale producers and consumers independently.

Durability is not infinite retention or automatic business correctness. Messages can still expire, exceed configured limits, or be deleted by policy. A consumer may perform an external side effect and crash before its acknowledgment reaches the server, causing the message to be processed again. Applications still need idempotent handlers, suitable retention, and monitoring.

How JetStream Works

A simplified message path is:

Publisher → NATS subject → matching stream → consumer delivery → acknowledgment

The publisher sends a message to a subject. The NATS server routes it to any stream whose subject filter matches. JetStream assigns stream sequence information and persists the message in memory or on disk. The server sends a publish acknowledgment after accepting it according to the stream and replica configuration.

A consumer tracks its delivery position and acknowledgment state separately from the publisher. It can request messages when ready (pull) or receive them as the server sends them (push). With explicit acknowledgments, unacknowledged messages can be retried after the acknowledgment timeout or configured backoff. This is generally an at-least-once model: a retry can result in duplicate processing.

JetStream also supports clustered storage. Stream replicas are placed on NATS servers and coordinated so the stream can remain available through some node failures. Replication helps with server availability, but it is not a backup against operator mistakes, unwanted publications, or retention policies that delete data.

JetStream and Kafka Compared

Both systems can retain messages and let consumers process them asynchronously, but their core data models differ.

The categories overlap: both can support durable workflows, fan-out, and replay. Kafka’s event-streaming architecture gives consumers independent progress through retained partitions. JetStream can be simpler when an application already uses NATS subjects and needs server-managed persistence without making partitions its primary scaling unit. Benchmark the actual message size, retention period, throughput, latency, and operational requirements.

Streams, Consumers, and Delivery Guarantees

Stream retention determines when stored messages are removed:

• Limits-based retention keeps messages until configured age, byte, or message limits are reached. It is useful when several consumer applications need independent access to a retained window.
• Interest-based retention removes messages after the relevant consumers have acknowledged them, subject to stream configuration.
• Work-queue retention removes messages after a consumer acknowledges them. It is intended for work distribution, not for multiple independent applications that each need to process every message.

Consumers define how messages are delivered and acknowledged. Durable consumers retain their server-side state across client disconnects and restarts; ephemeral consumers are suited to temporary consumption. Pull consumers let workers request bounded batches when capacity is available, while push consumers deliver messages to a configured subject and may use flow control. Consumer configuration should match the application’s concurrency, retry, and ordering requirements.

JetStream can detect duplicate publications when publishers provide a message ID within the configured duplicate window. Its exactly-once mechanisms combine publication deduplication with acknowledgment confirmation, but they do not make arbitrary external actions exactly once. A database write or HTTP request is outside the broker’s transaction boundary. Design those operations to tolerate retries; Kafka transaction semantics illustrate a similar boundary between broker guarantees and external side effects.

Conventional NATS client connections use TCP. RFC 9293 specifies TCP’s reliable, ordered byte-stream service, but that transport guarantee is not message persistence: JetStream’s stream storage and acknowledgments provide the application-level behavior.

Real-World Use Cases

JetStream is a fit for durable background work when a service should accept a task even if workers are temporarily offline. A payment service, for example, can publish a payment-authorized event and let separate fulfillment and notification consumers process it at their own pace. Handlers should use event IDs or another idempotency key so redelivery does not repeat an irreversible action.

It can also support device telemetry, service events, and operational pipelines where NATS subject routing is already in use. Independent durable consumers can process the same retained messages for different purposes, while work-queue retention can help distribute one task among workers. For broader asynchronous design patterns, see event-driven microservices.

JetStream may be less suitable when the workload depends on a long, partitioned history, a large Kafka ecosystem of connectors and stream processors, or a particular cross-partition ordering and replay model. Those are design requirements to test, not reasons to assume one broker is categorically faster or more reliable.

Getting Started with a Local JetStream Server

This single-node Docker Compose setup is for development. It persists data in a named volume, enables JetStream, and exposes the monitoring port locally. A single server is not a high-availability deployment.

services:
  nats:
    image: nats:2-alpine
    command: ["-js", "-sd", "/data", "-m", "8222"]
    ports:
      - "4222:4222"
      - "8222:8222"
    volumes:
      - nats-data:/data

volumes:
  nats-data:

Start the server, then install the official NATS CLI to manage streams and publish test messages:

docker compose up -d

nats --server nats://localhost:4222 stream add ORDERS \
  --subjects "orders.>" \
  --storage file \
  --retention limits \
  --max-age 24h \
  --replicas 1 \
  --defaults

nats --server nats://localhost:4222 pub orders.created \
  '{"order_id":"order-123","status":"created"}'

nats --server nats://localhost:4222 stream info ORDERS
nats --server nats://localhost:4222 stream view ORDERS

The stream stores messages on disk and accepts subjects beginning with orders.. stream info checks the stream configuration and message count; stream view lets you inspect the stored records. For a worker, create a durable consumer with an explicit acknowledgment policy and choose pull or push delivery based on backpressure needs. In production, configure authentication, TLS, resource limits, storage capacity, and multiple replicas across failure domains. Consult the JetStream documentation for deployment and consumer details.

Common Misconceptions

“JetStream guarantees every message is processed exactly once.”

Redelivery is possible when processing succeeds but acknowledgment is lost. Publisher deduplication and acknowledgment confirmation narrow duplicate cases, but downstream effects still need idempotency or their own transactional design.

“A durable consumer stores a separate copy of every message.”

The stream holds retained messages; a consumer primarily stores delivery configuration and progress. Multiple consumers can read the same stream without requiring one separately retained message log per consumer.

“JetStream and Kafka are interchangeable.”

They overlap, but subjects and stream consumers are not the same abstraction as partitioned logs and consumer groups. Ordering, replay, scaling, ecosystem integrations, and operational practices differ. Choose based on tested requirements rather than product labels.

Related Articles

• Compare the underlying models in event streaming with Apache Kafka.
• Learn how Kafka tracks parallel work in consumer groups and partition rebalancing.
• See where broker guarantees end in Kafka transactions and exactly-once processing.
• Explore the wider patterns in event-driven microservices.

Changelog

• Initial publication.