StorageTechnology

Block Storage Performance: Metrics, Benchmarks, and Tuning

Updated on
9 min read

Block storage performance is not a single speed rating. Database administrators, infrastructure engineers, and developers need to understand how request size, concurrency, latency, device limits, and the I/O path combine to affect an application. This guide explains those factors and gives a safe method for measuring a volume before changing its configuration.

What Is Block Storage Performance?

Block storage presents a device as addressable blocks that an operating system or application can read and write. The device may be a local SSD, a LUN on a storage array, or a cloud volume attached to a virtual machine. A filesystem can be layered on top, but the block device itself does not understand files or directories.

Performance describes how quickly that device completes a particular workload. A useful result must identify the operation, request size, read/write mix, concurrency, and measurement point. A volume’s advertised maximum IOPS, for example, does not predict the latency of a database transaction or the bandwidth available to a virtual machine.

Why Performance Depends on the Workload

A device can be busy without moving much data, or transfer large amounts of data with relatively few operations. Small random requests are common in transactional databases and may be limited by IOPS or latency. Large sequential requests, such as backups and media scans, are more likely to reach a bandwidth ceiling.

Applications also care about different parts of the latency distribution. Average latency can hide periodic stalls; p95 and p99 values help reveal slow requests that affect users or transaction deadlines. Reads and writes can behave differently because of device caching, RAID parity, replication, and durability requirements.

The target should therefore be an application-level service objective, not the highest benchmark number. Profile the actual workload first, then compare a controlled storage test with measurements from the filesystem, virtual machine, network, and storage service.

How Block Storage I/O Works

A typical request follows this path:

Application or database → filesystem or raw-device layer → operating-system block layer → device driver or virtual storage interface → local controller or storage network → volume or array → media

Completions travel back through the stack. Each layer can queue, combine, split, cache, or delay requests. On Linux, the multiqueue block layer (blk-mq) maps software submission queues to hardware dispatch queues so multiple CPUs can submit work concurrently. The exact path differs for a local NVMe namespace, a SCSI disk, an iSCSI LUN, NVMe over Fabrics, and a cloud volume. The Linux block multiqueue documentation describes the kernel’s queue model.

Several metrics help explain where a workload is constrained:

Metric What it measures How to interpret it
IOPS Completed read or write operations per second Meaningful only with request size, read/write mix, and concurrency
Throughput Data transferred per second Roughly IOPS multiplied by request size; large requests can reach bandwidth limits first
Latency Time for an I/O to complete Compare average and tail percentiles, and note whether the measurement includes guest, network, or application time
Queue depth I/Os waiting or in flight at a layer More outstanding work can increase utilization, but also increase waiting time and does not make the media faster

For example, 10,000 operations per second at 4 KiB per operation represents a very different workload from 10,000 operations per second at 256 KiB. The former transfers about 39 MiB/s; the latter is about 2.4 GiB/s before protocol and measurement overhead. Always read IOPS and throughput together with the test settings.

Queue depth is not one universal device setting. An application can submit concurrent requests, the kernel can maintain per-CPU and hardware queues, and a storage service can impose its own limit. A benchmark’s iodepth is the number of I/Os that benchmark attempts to keep outstanding; it is not a promise that the underlying device will run every request in parallel.

Components and Storage Variants

Storage option Typical strengths Constraints to measure Often suited to
HDD Low cost per capacity; efficient for large sequential transfers Seek latency, random IOPS, and rebuild behavior Archives and capacity-focused data
SATA or SAS SSD Lower access latency and stronger random I/O than HDDs Device endurance, controller limits, and shared-bus capacity General-purpose servers and mixed workloads
Local NVMe SSD High parallelism and bandwidth over PCIe with low device latency Thermal throttling, endurance, and host-local failure domain Latency-sensitive databases and scratch workloads
Network or cloud block volume Shared management, flexible capacity, and remote attachment Network path, service limits, burst behavior, and per-instance ceilings Virtual machines, managed databases, and clustered systems

These are broad tendencies, not guaranteed rankings. Firmware, device generation, request size, filesystem, host CPU, and service tier all matter. NVMe is a protocol and interface family, not a guarantee that an end-to-end workload will be faster than every SSD behind another interface.

In a cloud environment, check both the volume’s configured limits and the instance’s aggregate storage bandwidth and IOPS limits. Some services also vary performance with volume size or offer burst credits. The Amazon EBS volume type documentation is one example of why a volume type’s baseline and maximum rates must be read alongside attachment and instance constraints.

Real-World Workloads

  • Transactional databases: Often combine small random reads and writes with strict tail-latency and durability requirements. Include database logs and synchronous writes in the profile; a fast asynchronous benchmark may not represent commit latency.
  • Virtual machine boot and application volumes: Can produce bursty, mixed I/O as guests start services and users log in. Measure within the guest and at the host or storage service when possible.
  • Backups and analytics scans: Usually favor sustained sequential throughput and larger requests. Check whether the source, network, CPU, or destination is limiting the end-to-end transfer.
  • GPU and high-performance computing data paths: Storage latency and bandwidth can limit accelerator utilization even when GPU transfer itself is efficient. For that distinction, see GPU Direct Storage architecture.

The same volume can behave differently under each pattern. Do not infer database performance from a sequential bandwidth test or predict backup duration from a small-block random-read benchmark.

Practical Considerations: Measure and Tune Safely

Start by recording the device, filesystem, workload, and baseline. On Linux, lsblk shows the block-device layout, while iostat from the sysstat package can report extended device statistics:

lsblk -o NAME,TYPE,SIZE,ROTA,MODEL
findmnt /mnt/data
iostat -xz 1

In iostat, review read and write rates, request sizes, queue length, and wait times over a representative interval. Device utilization and wait fields are useful clues, not universal saturation tests: virtual and cloud devices can report values that do not correspond directly to a physical drive’s busy time. The iostat manual explains the reported fields and their limitations.

Use fio to generate a controlled workload. The following example first writes a dedicated test file, then measures random reads from it. Change the mount path and test-file name to a disposable file on the volume being evaluated:

mkdir -p /mnt/storage-bench

# This creates or overwrites only the dedicated test file.
fio --name=prepare \
  --filename=/mnt/storage-bench/fio-test.bin \
  --size=4G --rw=write --bs=1M \
  --ioengine=io_uring --iodepth=16 --direct=1 \
  --end_fsync=1 --group_reporting

fio --name=randread \
  --filename=/mnt/storage-bench/fio-test.bin \
  --size=4G --rw=randread --bs=4k \
  --ioengine=io_uring --iodepth=16 --direct=1 \
  --time_based --runtime=60 --group_reporting

This is only one workload sample. Confirm that the installed fio build supports io_uring; otherwise select an available engine documented by fio. Run tests in a lab or on an approved, isolated volume with enough free space. Never point a write test at a raw production device, filesystem metadata, or application data. direct=1 bypasses the operating-system page cache for this test, but does not eliminate every cache in a controller, hypervisor, or storage service.

For useful comparisons, keep the test file, request size, duration, filesystem, and host constant. Sweep concurrency and block sizes gradually, and test representative reads, writes, and mixed operations. Record average and tail latency, IOPS, throughput, CPU, and any network or service-side throttling. Compare with application measurements: a benchmark result is not an application service-level guarantee.

Tune only after locating the bottleneck. Check volume and instance limits, path redundancy, network congestion, filesystem behavior, CPU saturation, and competing workloads. Change one variable at a time and retain a way to roll back. Raising queue depth can help an underutilized device reach its throughput limit, but it can also increase latency and merely move the queue downstream. Avoid copying generic sysfs values or changing filesystem and RAID settings without workload-specific evidence.

For network-attached devices, verify the path and the storage protocol as well as the volume. iSCSI multipathing can improve path resilience and, on supported arrays, distribute I/O; it does not guarantee linearly higher bandwidth. Similarly, storage tiering should follow observed access patterns and recovery requirements rather than assumptions about a media label; see storage tiering implementation strategies.

Common Misconceptions

  • “The highest IOPS number means the fastest storage.” IOPS without request size, latency, and read/write mix is incomplete. A sequential workload may be constrained by bandwidth instead.
  • “More queue depth always improves performance.” Concurrency can keep a device busy, but excessive outstanding work increases waiting and tail latency. Find the useful point for the real workload.
  • “The operating system’s device utilization proves the physical drive is saturated.” Virtualization and cloud layers can obscure the physical device and its service-side queues. Correlate guest, host, network, and provider metrics.
  • “RAID or NVMe guarantees better performance.” RAID level, controller, failure mode, media, and workload determine the result. NVMe alone says little about the complete data path.
  • “A successful write is always durable on persistent media.” Completion and durability depend on flush or force-unit-access behavior and the guarantees of every cache and storage layer. Understand the filesystem, device, and service semantics before relying on a benchmark.
TBO Editorial

About the Author

TBO Editorial writes about the latest updates about products and services related to Technology, Business, Finance & Lifestyle. Do get in touch if you want to share any useful article with our community.