etcd Explained: Raft, MVCC, and Watches

When a Kubernetes control plane stores a Deployment or a node’s status, it needs more than a fast key-value database: every control-plane instance must agree on the committed state, and controllers need a reliable way to learn what changed. etcd provides that coordination layer. This explainer is for developers and platform operators who want to understand etcd’s Raft consensus, revisioned multi-version concurrency control (MVCC), watch API, and recovery boundaries.

Why etcd Is Being Discussed

etcd is easy to overlook because it usually runs behind an API server rather than as an application-facing database. Yet a cluster’s control plane depends on its availability and consistency. Kubernetes uses etcd as its backing store for cluster data, so quorum loss, storage pressure, or an unsafe restore can affect the ability to change and observe cluster state. The Kubernetes architecture documentation describes this role and calls out the need for a backup plan.

The topic matters beyond Kubernetes. Distributed applications need a consistent place to coordinate configuration, membership, leader ownership, or other small but important pieces of state. Understanding how etcd acknowledges writes and retains change history makes it easier to distinguish an application issue from a consensus, disk, or watch-recovery problem.

What Is etcd?

etcd is a distributed key-value store designed for highly available, strongly consistent data used by other distributed systems. A client sends operations such as reads, writes, transactions, and watches to the etcd API. A cluster of etcd members replicates the state so that a single surviving process is not the only copy of the authoritative data.

The etcd API documentation defines these operations as gRPC services, including key-value, watch, lease, cluster, and maintenance APIs. The API is not the consensus algorithm itself: clients interact with a logical keyspace, while members coordinate internally to agree on changes.

etcd is not a general-purpose relational database, a cache, or a full Kubernetes API server. It is most useful for relatively small, coordination-oriented records that need consistent updates and reliable change notification. Applications commonly access it through a service-specific client library rather than treating it as a public database endpoint.

The Problem It Solves

If several machines keep independent copies of configuration, they can disagree after a network interruption or a concurrent update. A leader-based service that stores data on just one machine avoids some conflicts, but that machine becomes a single point of failure. Replicating the data without an ordering protocol creates another problem: two members might accept different writes and later be unable to reconcile them safely.

etcd combines a replicated log with a key-value state machine. Members agree on the order of changes before applying those changes to their local stores. This gives clients one coherent view when the cluster has a quorum. When a majority is unavailable, etcd favors safety over accepting writes it cannot commit consistently. It does not make an arbitrary network partition disappear; distributed systems still have the failure boundaries described in our guide to network partitions and distributed system failures.

How Raft Replication Works

etcd uses Raft to replicate changes among its members. Raft organizes a cluster around a leader, which accepts proposals and coordinates replication, and followers, which replicate the leader’s log. If the leader becomes unavailable, eligible members hold an election and a new leader can be chosen when a majority can communicate.

A write follows a simplified sequence:

1. A client sends a request to an etcd endpoint.
2. The cluster leader proposes the change as a log entry.
3. Members persist and acknowledge the entry.
4. Once a majority acknowledges it, the entry is committed and applied to the key-value state.
5. The client receives a response with metadata such as the store revision.

Consensus therefore establishes a shared order for writes, rather than merely copying bytes between machines. The Raft project introduces the replicated-state-machine model and links to the original paper, “In Search of an Understandable Consensus Algorithm”.

A cluster of three members can tolerate one unavailable member; a cluster of five can tolerate two. Losing a majority prevents new writes from being committed. Adding members does not always improve resilience: a four-member cluster still needs three members for a majority and tolerates only one failure, just like a three-member cluster. Operators commonly choose an odd member count and place members across independent failure domains, while accounting for network latency and disk performance.

The client-facing API uses gRPC, while Raft handles the internal agreement on state changes. The IETF’s RFC 9113 standardizes HTTP/2, the transport foundation used by gRPC. That transport detail is distinct from Raft’s quorum and log rules: a healthy client connection alone does not prove that the cluster can commit a write.

Key Concepts: Revisions, Watches, and Leases

etcd combines several related mechanisms. Keeping their roles separate helps make operational behavior easier to reason about.

These are not interchangeable counters or guarantees. In particular, the Raft log index and the MVCC store revision describe different layers of the system.

MVCC and Store Revisions

etcd’s keyspace is a flat, ordered set of byte-string keys. Each committed mutation advances a cluster-wide store revision; a transaction that changes several keys can apply them atomically at one revision. Key metadata includes fields such as its creation revision, modification revision, per-key version, and attached lease. The etcd data model guide explains how revisions provide point-in-time reads and preserve previous versions.

This history lets a client read a prior revision or start a watch from a known position. It is not retained forever. Compaction removes revisions older than the configured history window so the database does not grow without bound. If a client asks for a revision that has already been compacted, etcd cannot replay it; the client must read a fresh consistent view and resume watching from that newer revision.

Watches and Leases

A watch streams matching key changes from a requested revision. A controller can use a range read to establish its initial state, then watch subsequent revisions instead of continuously polling. If it disconnects, it can resume from the last revision it processed, provided that revision remains available. When it has been compacted, the client must resynchronize rather than assume it received every event.

Leases attach a bounded lifetime to keys. A client periodically sends keep-alives; if the lease expires, keys attached to it are removed. Leases are useful for temporary registrations and coordination, but an expired lease cannot stop a paused process from resuming and issuing work. For that reason, a lock may need a downstream fencing check as described in distributed locks, leases, and fencing tokens.

Real-World Uses

• Kubernetes cluster state: The API server stores and reads cluster objects through etcd; controllers observe changes and reconcile actual state toward the declared state.
• Service discovery and configuration: Services can publish small configuration or endpoint records and watch for updates, provided they have a defined ownership and expiry policy.
• Leader election and distributed locks: Applications can use transactional comparisons, leases, and revisions to coordinate ownership. The protected resource still needs appropriate safeguards against stale work.
• Control-plane coordination: Distributed systems can store membership or other metadata that must be consistent across a small group of servers.

These uses share an important constraint: etcd is intended for coordination data, not for unbounded application payloads or high-volume event history. Retention, key size, request rate, and recovery objectives should be part of the design.

Getting Started: Write, Watch, and Check a Local Cluster

The etcd project’s release instructions include downloadable binaries and a container-based setup. The following launches a single-member local demonstration. It binds the client and peer ports to loopback so they are not exposed to the local network:

docker run --rm -d --name etcd-demo \
  -p 127.0.0.1:2379:2379 \
  -p 127.0.0.1:2380:2380 \
  gcr.io/etcd-development/etcd:v3.7.3 \
  /usr/local/bin/etcd \
  --name s1 \
  --data-dir /etcd-data \
  --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://127.0.0.1:2379 \
  --listen-peer-urls http://0.0.0.0:2380 \
  --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --initial-cluster s1=http://127.0.0.1:2380 \
  --initial-cluster-token etcd-demo \
  --initial-cluster-state new

Check health, write a key, and read it back:

docker exec etcd-demo etcdctl --endpoints=http://127.0.0.1:2379 endpoint health
docker exec etcd-demo etcdctl --endpoints=http://127.0.0.1:2379 put /demo/greeting hello
docker exec etcd-demo etcdctl --endpoints=http://127.0.0.1:2379 get /demo/greeting
docker exec etcd-demo etcdctl --endpoints=http://127.0.0.1:2379 endpoint status --write-out=table

To see changes, start this watch in one terminal and update the key from another:

docker exec -it etcd-demo etcdctl --endpoints=http://127.0.0.1:2379 watch /demo/ --prefix

Create a snapshot and copy it out of the container before removing the demo:

docker exec etcd-demo etcdctl --endpoints=http://127.0.0.1:2379 snapshot save /tmp/etcd-snapshot.db
docker cp etcd-demo:/tmp/etcd-snapshot.db ./etcd-snapshot.db

The example is intentionally single-member and uses plaintext HTTP only on loopback; it is not a production deployment template. Production clusters need authenticated TLS, restricted client and peer access, durable storage, quorum-aware placement, tested snapshots, and alerts for member health, disk usage, and quota alarms. Kubernetes operators can also use the cluster monitoring guide to place etcd signals alongside API-server and node health.

Recovery and Common Misconceptions

A snapshot is a recovery point, not a substitute for a tested restore plan. The official etcd disaster recovery guide describes saving a snapshot, checking its status, and restoring it into new data directories. Restoring a Kubernetes-backed store to an older revision can leave controller caches out of sync. The guide explains revision bumps and marking revisions compacted so watching clients rebuild their caches. Treat restore as a cluster-level procedure: validate the snapshot, coordinate API-server access, and follow the deployment’s recovery runbook.

The maintenance guide also distinguishes history compaction from defragmentation. Compaction makes old revisions unavailable and frees them for reuse inside the backend; defragmentation reclaims backend space for the filesystem. Neither operation should be triggered blindly during an incident.

“Three members mean the cluster always works if one server is reachable.” A member by itself is not a quorum. A three-member cluster can generally make progress with two members, not one.

“A successful local read proves the cluster is healthy.” A serializable read can be served locally and may be stale. Linearizable reads and writes have stronger coordination requirements, and a member’s process health does not prove the cluster can commit.

“A watch is a permanent event log.” Watches expose retained keyspace history, bounded by compaction. Clients need a recovery path for compacted revisions, typically a fresh range read followed by a watch from the returned revision.

“A snapshot can be copied back over a running member.” Snapshot restore creates new data directories and a new logical cluster identity. Follow the documented restore and membership procedure rather than replacing files in a live data directory.

Related Articles

• Kubernetes architecture explained places etcd in the wider control plane.
• Distributed system failures and network partitions explains why quorum loss changes what a system can safely do.
• Distributed locks, leases, and fencing tokens covers lease-based coordination and stale owners.
• Kubernetes cluster monitoring covers control-plane health signals.

Changelog and Last Updated

• First publication.