Ceph on Proxmox VE: Architecture and Setup

Ceph on Proxmox VE combines a virtualization cluster with distributed storage managed from the same platform. It is useful when virtual machines need shared storage for migration or high availability, but the design also makes network, disk, and failure planning part of the hypervisor’s operation. This guide explains how the integration works, how it differs from a standalone Ceph deployment, and what to verify before putting VM disks on it.

What Is Ceph on Proxmox VE?

Ceph is a distributed storage system that spreads data across storage daemons on multiple servers. Proxmox VE provides a supported way to install and configure Ceph alongside its virtual machines and containers, using the cluster’s web interface and Proxmox-specific command-line tools. The Proxmox VE overview describes the platform’s integrated cluster, storage, and virtualization management; its administration guide documents the Ceph workflow.

The result is often called hyper-converged infrastructure: the same Proxmox nodes run guest workloads and contribute disks to a Ceph cluster. A VM can store its virtual disk on a Ceph RADOS Block Device (RBD) pool, which is available to multiple Proxmox nodes. Ceph can also provide CephFS or an S3-compatible API through its RADOS Gateway, but these interfaces have different uses and are not automatically created by adding an RBD pool.

Why Run Ceph with Proxmox?

Local VM storage is simple, but a virtual disk usually belongs to one host. Moving a VM to another node then requires copying its disk or using a separate shared-storage system. Proxmox VE clustering coordinates nodes and their guests; Ceph adds a distributed storage layer that can make selected VM disks available across those nodes.

That shared layer supports operational patterns such as moving a guest without first copying its entire disk and restarting a guest on another node after a host failure. Those outcomes depend on a healthy Ceph cluster, a suitable pool policy, and correctly configured Proxmox HA. Ceph does not itself migrate or restart every VM, and neither feature removes the need for backups.

The trade-off is a larger failure domain and more operational work. A node may be serving VM CPU and memory while also handling client I/O, replication, and recovery for Ceph. A failed disk can trigger recovery traffic that competes with application traffic. Planning for spare capacity, separate network paths where appropriate, and monitoring is therefore part of the deployment rather than later tuning.

How the Integration Works

Proxmox VE and Ceph are related but distinct clusters. Proxmox VE uses Corosync for membership and quorum; Ceph uses monitors (MONs) to maintain its own maps and quorum. Managers (MGRs) provide management and monitoring functions, while object storage daemons (OSDs) store data and handle recovery. A Proxmox cluster does not make Ceph highly available by itself, and Ceph monitor quorum does not replace Proxmox quorum.

When a VM writes to an RBD disk, the client uses the Ceph cluster map to find the placement group and OSDs responsible for the data. The CRUSH algorithm maps data to OSDs using the configured topology and rules. A rule can separate copies across hosts or other failure domains; the Ceph CRUSH documentation explains this placement model. The configured replica count is only useful if the cluster has enough independent OSD hosts to satisfy it.

Proxmox’s pveceph tools handle the integrated lifecycle; they are not interchangeable with a generic cephadm tutorial. Existing Ceph clusters can also be connected to Proxmox as external storage, which is useful when compute and storage are operated separately.

Components and Design Choices

Hosts, monitors, managers, and OSDs

Proxmox’s guidance recommends at least three nodes and twelve OSDs, distributed evenly, for a Ceph cluster. Treat that as a planning recommendation, not a guarantee that every workload or failure scenario will be safe. Small labs can test the tools with fewer resources, but they should not be mistaken for a production failure-tolerant design. Three monitors are commonly used for quorum, and redundant managers help keep management functions available.

Use dedicated, direct-access disks for OSDs rather than placing Ceph on top of a hardware RAID volume. An OSD setup prepares its selected device and can destroy existing data; verify the device identity and contents before adding it. Keep the Proxmox operating system and unrelated guest data off disks reserved for Ceph.

Networks and placement

Proxmox recommends a fast network reserved for Ceph traffic, with at least 10 Gbps as a baseline in its guidance. Actual needs depend on guest traffic, disk count, recovery behavior, and whether client and replication traffic share links. A dedicated Ceph network is not useful if its switches, uplinks, or host interfaces are oversubscribed. Measure the complete path and keep management access available when the storage network is degraded.

Plan failure domains before creating pools. A replication size of three requires copies to be placed on three eligible failure domains; it does not mean that any three disks in a single server provide host-level protection. min_size also affects availability: if too few replicas remain, writes can stop rather than continue unsafely. Understand the effects of losing a host before choosing pool settings.

RBD, CephFS, and RGW

RBD is the usual Ceph interface for Proxmox VM disks. CephFS provides a shared filesystem when applications need file and directory semantics. RADOS Gateway provides object APIs for applications built for S3 or Swift. A deployment should enable only the interfaces it needs; one pool or one management screen does not make these services equivalent.

Real-World Uses

Ceph on Proxmox is a fit for a virtualization cluster that needs shared VM storage and has enough nodes, disks, network capacity, and operational attention to support it. A home lab can use it to learn distributed placement, recovery, and cluster behavior, provided important data is backed up elsewhere. A small organization may use it to keep VM storage accessible across compute nodes without a separate storage array.

It is not automatically the right choice for a single hypervisor, a small number of VMs, or a workload that only needs backup retention. Local ZFS or LVM storage can be simpler for one host. A standalone NAS or external Ceph cluster can make sense when storage should have a separate lifecycle from virtualization. The Proxmox Backup Server explainer covers a dedicated backup system, which solves a different problem from shared primary storage.

Setting Up Ceph on Proxmox VE

Prepare the PVE cluster and hardware

Install Proxmox VE on the intended nodes and form the PVE cluster before configuring Ceph. Confirm node names resolve consistently, time is synchronized, and all nodes can communicate over the planned management and storage networks. Decide which interfaces carry Corosync, VM traffic, migrations, and Ceph traffic; one physical network may carry more than one role, but that increases contention and common failure risks.

Inventory the disks on every node and identify which dedicated devices can be erased for OSD use. Check that the nodes have enough memory and CPU for both guests and storage services, reserve network capacity for recovery, and leave free space for rebalancing. Proxmox’s Ceph administration guide should be treated as the source of truth for supported Ceph releases and version-specific steps.

Install and initialize Ceph

Use the Proxmox web interface’s node-level Ceph section and installation wizard to select a supported Ceph release and configure the cluster. The corresponding CLI workflow begins with installing Ceph on each participating node, then creating shared configuration with the dedicated Ceph network:

# Run on each participating Proxmox VE node
pveceph install

# Run once from a cluster node; replace the CIDR with your Ceph network
pveceph init --network 10.10.10.0/24

The network in pveceph init is an example only. Use the actual subnet configured for Ceph, and verify routing and firewall rules before proceeding. Proxmox distributes its Ceph configuration to the PVE nodes through its cluster configuration filesystem.

Create services, OSDs, and an RBD pool

Create monitors on the planned nodes and managers using the Ceph section of the PVE interface or the pveceph commands. Add OSDs only from the dedicated disks that have been positively identified. The Proxmox wizard guides disk selection and pool creation; review each device and pool policy rather than accepting defaults without checking that the node count and failure domains can satisfy them.

For VM storage, create an RBD pool and register it as Proxmox storage. Check the pool’s replica size and min_size, placement rule, and autoscaler state. Then create a small test VM disk on that storage and confirm it is visible from the intended PVE nodes. Do not use ceph-volume ... zap, disk wipe commands, or other destructive steps unless the target device has been verified and its data is disposable.

Verify health and test recovery

After installation and whenever making a topology change, check both Ceph and Proxmox storage status:

ceph -s
ceph health detail
pvesm status

Investigate degraded placement groups, down OSDs, insufficient free space, or a pool that is not active before placing important guests on it. In a non-production test, simulate a planned OSD or host outage and observe whether the pool remains available and how long recovery takes. Do not pull disks or power off production nodes as an informal test.

Proxmox also exposes an API for repeatable management tasks. Its API viewer documents available endpoints; when clients call those endpoints over HTTP, request and response semantics are defined by the IETF’s HTTP Semantics specification, RFC 9110. Automation should check task results and cluster health instead of treating an accepted API request as proof that data is healthy.

Common Misconceptions

“Ceph is a checkbox that makes local disks shared.”

Ceph is a distributed system with independent quorum, placement, recovery, and capacity requirements. The PVE interface simplifies management but does not remove the need to design failure domains and monitor the cluster.

“Three copies mean three-node availability.”

Three copies are only placed across distinct hosts if the CRUSH rule and available topology require it. A pool can lose redundancy when hosts fail, and I/O can stop if the number of remaining replicas falls below min_size. Review those settings alongside the actual OSD layout.

“Ceph replaces backups.”

Ceph can keep storage available through configured disk or host failures, but it can also faithfully store deletion, corruption, or unwanted changes. Keep independent, versioned backups and test restores. Shared storage improves access to VM disks; it does not provide a historical recovery copy by itself.

“A healthy dashboard means the system is production-ready.”

An HEALTH_OK status is a point-in-time cluster signal. It does not prove that capacity is sufficient, recovery meets a service objective, credentials are protected, or a guest can be restored. Use monitoring, alerting, documented maintenance procedures, and planned restore tests.

Related Articles

• Ceph storage cluster architecture and deployment explains Ceph services and a separate cephadm workflow.
• Proxmox VE, ESXi, and XCP-ng compared compares virtualization platforms and their storage models.
• Proxmox Backup Server explained covers versioned guest backups and restore testing.
• Erasure coding versus replication explains storage protection trade-offs.

Changelog

Last updated: 2026-09-25.

• 2026-09-25: Initial publication.