StorageLinux

Ceph Storage Cluster Deployment: Architecture and Practical Beginner's Guide

Updated on
15 min read

Ceph is a distributed storage platform for teams that need resilient storage without buying a single proprietary array. It can expose block devices to virtual machines, an S3-compatible API to applications, and a POSIX-like filesystem to users while storing the data across ordinary servers. This guide explains the architecture behind a Ceph storage cluster and shows a safe starting workflow for deploying one with cephadm.

The examples are suitable for a lab or a small proof of concept. A production cluster also needs capacity modeling, failure-domain design, monitoring, tested recovery procedures, and a backup or disaster-recovery plan. Ceph can keep serving data when a disk or host fails, but replication is not a substitute for an independent backup.

What Is a Ceph Storage Cluster?

Ceph is an open-source, scale-out storage system built on RADOS (Reliable Autonomic Distributed Object Store). Instead of placing data behind one controller, it spreads data across Object Storage Daemons (OSDs) and uses a cluster map plus the CRUSH placement algorithm to calculate where objects belong.

Applications usually do not write directly to a disk. They use one of Ceph’s storage interfaces:

  • RBD (RADOS Block Device): Presents virtual block devices for virtual machines, databases, and container persistent volumes.
  • CephFS: Provides a distributed filesystem. Metadata servers manage directories and file metadata, while file contents remain in RADOS.
  • RADOS Gateway (RGW): Provides S3-compatible and Swift-compatible object APIs for applications and backups.

This unified model is the main reason to choose Ceph. One cluster can provide several storage protocols, but every additional protocol also adds components to operate. A small environment that only needs one shared filesystem may be better served by a simpler system.

Why Deploy Ceph?

Traditional storage arrays make scaling and failure recovery dependent on a controller pair, a vendor-specific chassis, or a licensed feature. A Ceph cluster moves those responsibilities into software running across multiple hosts:

  • Scale out: Add OSD hosts or drives instead of replacing one central array.
  • Failure tolerance: Replication or erasure coding keeps data available when configured failure domains fail.
  • Direct client access: Clients use cluster maps to reach the responsible OSDs rather than sending every I/O through a gateway.
  • Unified protocols: Block, file, and object access can share the same RADOS foundation.
  • Automated recovery: Ceph detects failures, marks data degraded, and re-creates missing copies or shards when resources are available.

These benefits come with network, CPU, memory, and operational costs. Recovery traffic competes with application traffic, and a cluster that is nearly full has less room to recover. Plan for headroom rather than treating all raw disk capacity as usable capacity.

How Ceph Works: Architecture and Data Flow

A useful mental model is:

Client -> cluster map from MONs -> CRUSH placement -> pool and placement group -> OSDs -> replication or erasure coding -> health and recovery

Core daemons

Component Responsibility When it is needed
MON Maintains authoritative cluster maps and quorum, including the monitor, OSD, and placement state. Every Ceph cluster; use an odd quorum-sized set such as three monitors for a small production cluster.
MGR Runs management modules, metrics, the dashboard, and orchestration integration. Every modern deployment should have redundant managers.
OSD Stores data, serves client I/O, participates in recovery, and reports device health. Every disk or logical storage device used for data.
MDS Tracks CephFS namespaces, directories, and metadata operations. Only when the cluster provides CephFS.
RGW Translates S3 or Swift requests into RADOS operations. Only when applications need object APIs.

MONs coordinate cluster state; they do not store the normal application payload. MGRs provide management and telemetry; they are not a replacement for MON quorum. OSDs are where capacity and most storage I/O live, so an installation with three MONs but only one storage host is not resilient to a host failure.

Pools, objects, and placement groups

Ceph presents applications with pools. RBD images, RGW objects, and CephFS data are broken into RADOS objects and stored in a pool. A pool has a data-protection policy, a CRUSH rule, and a number of placement groups (PGs).

PGs are logical groups used to organize objects and distribute work across OSDs. They are not physical directories and are not a direct promise of a specific number of disks. The PG autoscaler can recommend or manage PG counts as the cluster changes, while administrators still need to review warnings and workload behavior. An arbitrary large PG count can waste memory and create recovery overhead.

CRUSH and failure domains

CRUSH calculates placement from the cluster topology and policy. A CRUSH rule can require replicas or shards to land on different OSDs, hosts, racks, or sites. That distinction matters: three copies on three disks in one server do not protect against loss of the server.

When an OSD or host fails, Ceph temporarily marks the affected PGs degraded or misplaced. It then selects replacement OSDs according to the CRUSH rule and rebuilds the missing replicas or fragments. Recovery consumes disk and network bandwidth, so recovery priorities and throttles should be tested with the expected workload rather than blindly maximized.

Storage media and BlueStore

An OSD commonly uses a dedicated block device with BlueStore, Ceph’s storage backend. BlueStore writes data directly to the device and keeps metadata in its internal key-value database. Fast devices can optionally host the block.db or block.wal areas for workloads that benefit from lower metadata or write latency, but adding a fast device does not make a slow network or undersized cluster fast.

For an authoritative description of the current components and data path, see the Ceph architecture documentation. The CRUSH map documentation explains how topology-aware placement works.

Ceph Storage Interfaces and Protection Choices

The interface should follow the application rather than the other way around:

Interface or policy Best fit Important trade-off
RBD VM disks, databases, Kubernetes persistent volumes, and other block workloads Low-level block semantics require the guest or filesystem to handle consistency and backups.
CephFS Shared POSIX files, home directories, build workspaces, and content repositories MDS sizing and metadata workload matter, especially with many small files.
RGW Object data, backups, logs, media, and applications designed for S3 Object APIs are not a drop-in replacement for a POSIX filesystem.
Replicated pool Hot block data, metadata, and latency-sensitive workloads Simple recovery and good read behavior, but raw capacity overhead is commonly two extra copies for a size of three.
Erasure-coded pool Large objects, backups, and colder data Better usable capacity, with more CPU, network, and recovery complexity; not every workload can use one directly.

Replication stores complete copies and is usually the easiest first protection policy. Erasure coding stores k data chunks and m coding chunks, so the raw storage factor is (k + m) / k. It can reduce overhead, but small writes and degraded reads may require read-modify-write or reconstruction. Use a replicated pool for critical metadata and latency-sensitive workloads unless the interface and workload have been tested with erasure coding.

The Ceph RBD documentation and CephFS documentation cover the client-specific behavior. For a broader explanation of the durability trade-off, see our erasure coding versus replication guide.

Planning a Ceph Deployment

Hosts, disks, and network

For a resilient small cluster, plan at least three hosts and place monitors across those hosts. Keep OSD capacity distributed across failure domains. A lab can run several daemons on fewer machines, but that demonstrates commands rather than high availability.

Before installing Ceph:

  1. Choose stable hostnames and make forward and reverse name resolution work between nodes. The DNS configuration guide for Linux covers the resolver side of this preparation.
  2. Synchronize clocks on every host. Consensus and troubleshooting become unreliable when system time drifts.
  3. Use a reliable network with enough bandwidth for client I/O, replication, and recovery at the same time. Separate client and cluster networks only when the topology and workload justify the operational cost.
  4. Inventory disks with ceph orch device ls. Do not assume an empty-looking device is safe to erase; verify its identity and intended use.
  5. Reserve CPU, RAM, and SSD or NVMe capacity for OSD metadata where the workload benefits from it. Leave free capacity for recovery and expansion.
  6. Define failure domains before creating pools. Decide whether the cluster must survive a disk, host, rack, or site failure.

Cephadm manages Ceph daemons as containers and uses an orchestrator to place them. Use the official cephadm deployment documentation for the supported operating system, container image, and release instructions rather than copying an old package repository command.

A current Ceph deployment workflow

The following is a lab-oriented outline. Replace the release placeholder with a supported release from the official documentation, and use a real monitor address reachable by the other hosts.

1. Install cephadm and bootstrap the first host

export CEPH_RELEASE="<supported-release>"
curl --silent --remote-name --location \
  "https://github.com/ceph/ceph/raw/${CEPH_RELEASE}/src/cephadm/cephadm"
chmod +x cephadm
sudo ./cephadm add-repo --release "${CEPH_RELEASE}"
sudo ./cephadm install

# Bootstrap creates the first MON and MGR and enables orchestrator access.
sudo cephadm bootstrap --mon-ip 10.0.0.11

# Run the Ceph CLI from the cephadm-managed environment.
sudo cephadm shell -- ceph -s

Bootstrap prints the dashboard address and creates the SSH material used to manage additional hosts. Protect the generated administrator key and dashboard credentials; do not paste them into source control or shared shell history.

2. Add hosts and verify their devices

Copy the orchestrator’s public key to each additional host, then register the host with the cluster:

sudo ceph cephadm get-pub-key > ceph.pub
ssh-copy-id -f -i ceph.pub root@node2
ssh-copy-id -f -i ceph.pub root@node3

sudo ceph orch host add node2 10.0.0.12
sudo ceph orch host add node3 10.0.0.13
sudo ceph orch host ls
sudo ceph orch device ls

The device list distinguishes available devices from devices already used by a filesystem or another OSD. Treat any command that zaps or prepares a device as destructive and validate the device path first.

3. Place monitors, managers, and OSDs

Use the orchestrator to spread control-plane daemons and add only the intended storage devices:

# The exact placement can be expressed by host labels in a larger cluster.
sudo ceph orch apply mon --placement="3 node1 node2 node3"
sudo ceph orch apply mgr --placement="2 node1 node2"

# Prefer an explicit device while learning or when a host has mixed-purpose disks.
sudo ceph orch daemon add osd node2:/dev/sdb
sudo ceph orch daemon add osd node3:/dev/sdb

sudo ceph orch ps
sudo ceph -s

For homogeneous storage hosts, ceph orch apply osd --all-available-devices can manage all devices that Ceph identifies as available. Review ceph orch device ls first. Do not use that option on a host containing disks that must remain untouched.

4. Create a small RBD test pool

Create a test pool only after OSDs are healthy. The PG value below is intentionally small for a lab; use autoscaler guidance and the placement group documentation for a real workload.

sudo ceph osd pool create rbdpool 32
sudo ceph osd pool application enable rbdpool rbd
sudo rbd pool init rbdpool

sudo rbd create rbdpool/test-image --size 10240
sudo rbd info rbdpool/test-image

Before serving real data, set a protection policy that matches the number of hosts and the failure domain. A pool configured for three copies cannot safely place three host-separated copies if the cluster has only two suitable hosts. Validate the resulting CRUSH rule, pool settings, and available capacity.

5. Validate health and client access

Use the CLI and dashboard together. The CLI is easier to automate and troubleshoot; the dashboard is useful for trends, topology, and operator workflows. The Ceph dashboard documentation describes the available modules.

sudo ceph -s
sudo ceph health detail
sudo ceph osd tree
sudo ceph osd df tree
sudo ceph df
sudo ceph orch ps

Do not stop at HEALTH_OK immediately after bootstrap. Create and remove a test object, inspect placement, simulate a controlled OSD outage in a non-production environment, and confirm that recovery completes. Record the commands and expected alerts so an on-call engineer can recognize a real failure.

Operating and Scaling a Ceph Cluster

Capacity and recovery

Monitor raw and usable capacity, PG states, OSD fullness ratios, recovery queues, client latency, and network saturation. A nearly full cluster may refuse writes or have too little space to rebuild a failed OSD. Add capacity before emergency thresholds, and rebalance gradually so recovery does not starve application traffic.

When adding OSDs, check that the CRUSH hierarchy places data across the intended hosts and racks. When removing one, use the orchestrator’s documented removal workflow and wait for data to migrate before physically reusing the disk. Never pull a disk from a healthy OSD simply because it appears busy.

Security

Ceph uses keyrings and capability-scoped identities. Give clients access only to the pools and operations they require, protect the administrator key, and rotate credentials through a documented process. Restrict dashboard access, use TLS where appropriate, segment management traffic, and apply host security updates without bypassing the supported Ceph upgrade path.

For Kubernetes, use the Rook operator and CSI drivers when the cluster lifecycle is intentionally managed by Kubernetes. A Rook deployment is not just a different install command: it changes ownership, upgrades, secrets, and failure handling. Our container storage guide explains how Ceph fits among CSI-backed storage choices.

Backups and upgrades

Snapshots and clones are useful for testing and short-term recovery, but they remain inside the same failure domain unless copied elsewhere. Back up application data and configuration to an independent target, and test restoring a representative workload.

Follow the Ceph upgrade documentation for supported release transitions. Keep the cluster healthy before an upgrade, confirm client compatibility, upgrade in the documented order, and monitor recovery between stages. Do not mix an unplanned operating-system migration with a Ceph upgrade during an incident.

Common Failure Modes and Troubleshooting

Symptom Likely area First checks
HEALTH_WARN after a disk outage Degraded PGs or an OSD state change ceph health detail, ceph osd tree, and the affected daemon logs.
OSDs flap between up and down Network, power, media, or resource pressure Host reachability, clock sync, disk health, latency, and ceph orch ps.
Slow requests during recovery Recovery traffic competing with clients Client latency, network utilization, OSD queues, and recovery settings; do not simply disable recovery.
Uneven fullness CRUSH topology, device classes, or pool rules ceph osd df tree, CRUSH hierarchy, pool rules, and whether new OSDs are actually in the same class.
MON quorum lost Network partition or too few surviving monitors ceph quorum_status, host connectivity, and monitor logs. Restore quorum before making topology changes.
CephFS metadata latency MDS load or a metadata-heavy workload MDS health, active sessions, directory hot spots, and the CephFS client workload.

Change one variable at a time and capture the cluster state before and after the change. ceph health detail usually identifies the affected subsystem, but the remedy should follow the corresponding official runbook rather than a copied command from an unrelated release.

Common Misconceptions

“Three monitors provide three copies of my data.”

They do not. MONs provide quorum and cluster state. OSDs store application data. Data protection comes from the pool’s replication or erasure-coding policy and its CRUSH failure domain.

“Ceph replaces backups.”

Ceph protects against the failures its policy covers, such as an OSD or host failure. It does not automatically protect against accidental deletion, compromised credentials, application corruption, or a mistake replicated to every copy. Keep independent backups and test recovery.

“Adding more disks always makes Ceph faster.”

More OSDs can increase parallelism, but network bandwidth, CPU, recovery traffic, pool layout, device latency, and client behavior can become the bottleneck. Benchmark the workload and verify that the new devices are placed in the intended CRUSH hierarchy.

“CephFS, RBD, and RGW are interchangeable.”

They share RADOS, but their client semantics differ. A block device expects a filesystem or database to manage blocks; CephFS provides file and directory semantics; RGW provides an object API. Select the interface that matches the application.

“HEALTH_OK means the design is production-ready.”

Health is a current cluster signal, not a capacity plan, security review, backup test, or disaster-recovery exercise. Production readiness includes failure-domain validation, alerting, runbooks, upgrades, and restore tests.

TBO Editorial

About the Author

TBO Editorial writes about the latest updates about products and services related to Technology, Business, Finance & Lifestyle. Do get in touch if you want to share any useful article with our community.