iSCSI SAN Implementation Guide

iSCSI SAN (Internet Small Computer Systems Interface over TCP/IP) provides block-level storage over standard Ethernet. This guide is for systems engineers, Linux sysadmins, SREs and storage architects who need a repeatable, secure process to design, deploy and operate iSCSI SANs. It focuses on Linux-first tools and vendor-agnostic patterns that work across small clusters and production arrays.

What is iSCSI SAN?

iSCSI maps the SCSI command set to TCP/IP so remote initiators can perform block I/O against LUNs on a target. The protocol (IETF RFC 3720) defines initiator/target roles, login and session phases, discovery models and security options (CHAP, IPsec) — use it as the compatibility and security reference: https://datatracker.ietf.org/doc/html/rfc3720.

Core building blocks

• IQN (iSCSI Qualified Name): unique identifier for initiators and targets
• Portal / portal group (IP:port): address where a target listens (default port 3260)
• Target Portal Group (TPG): logical grouping of portals, ACLs and access controls
• LUN / backstore: the on-disk object presented to clients (fileio, block device, LVM logical volume, ZVOL)

The Problem iSCSI Solves

iSCSI lets organizations consolidate block storage without Fibre Channel hardware, using familiar Ethernet infrastructure. It enables shared VM datastores, centralizes backups and simplifies management for clusters that require block semantics (fsync, raw disks). The trade-offs are network design, predictable latency and security—these must be planned for production usage.

How it Works / Architecture

iSCSI establishes sessions after discovery and login: the initiator negotiates parameters, enters Full Feature Phase and issues SCSI commands transported over TCP. Key operational details:

• Transport: TCP (default 3260). Optionally use iSER (iSCSI Extensions for RDMA) for lower CPU overhead on RDMA-capable networks.
• Sequencing: CmdSN/ExpCmdSN and task IDs ensure ordered command processing and loss-recovery behavior.
• Multipath: expose multiple portals across NICs/switches and use DM‑Multipath on initiators for path failover and load-balancing.

Architectural patterns

• Single-host targets for lab/small infra
• Multi‑portal production targets with dedicated storage network (VLAN/VRF), separate management plane
• Array-style designs using LIO or vendor arrays with ALUA for path prioritization and asymmetric active/optimized paths

Important implementation choices

• Target implementation (LIO vs tgt vs SCST) — kernel vs userspace, virtualization hooks (vhost), enterprise features
• Backstore type — raw block (best performance), LVM/ZVOL for snapshotting, fileio for simple testing
• Pathing & redundancy — number of portals, NIC teaming, multipath policy (queue-length, rr, service-time)
• Security model — CHAP alone vs CHAP+IPsec vs physically segregated storage networks

Components / Key Concepts

• Initiator: open-iscsi provides iscsid (daemon) and iscsiadm (admin) for discovery, login, iface binding and persistent node records: https://github.com/open-iscsi/open-iscsi
• Target implementations:
  • LIO (targetcli) — in-kernel, full feature set, integrates with vhost and virtualization
  • tgt — userspace, simpler for small deployments
  • SCST — kernel modules often used by vendors for optimized paths
• Multipathing: DM‑Multipath with appropriate /etc/multipath.conf policies; ALUA compatibility improves path selection for arrays
• Performance offloads: TSO/GRO/LSO, checksum offload, and RSS for distribution; iSER and RDMA offloads when available

Real-World Use Cases

• VM datastores for KVM/QEMU where block-level semantics and snapshots are needed
• Consolidated block storage for database servers with multipath for redundancy
• Boot-from-iSCSI for stateless or diskless servers in scale-out deployments

Getting Started / Practical Guide

Install and enable initiator/target tools (examples):

# Debian/Ubuntu
sudo apt update
sudo apt install -y open-iscsi targetcli-fb
sudo systemctl enable --now iscsid

# RHEL/CentOS
sudo dnf install -y iscsi-initiator-utils targetcli
sudo systemctl enable --now iscsid

Create and export a file-backed LUN with targetcli (target node):

sudo mkdir -p /var/iscsi_disks
sudo dd if=/dev/zero of=/var/iscsi_disks/disk1.img bs=1M count=10240
sudo targetcli
/backstores/fileio create disk1 /var/iscsi_disks/disk1.img 10G
/iscsi create iqn.2026-04.com.example:target1
/iscsi/iqn.2026-04.com.example:target1/tpg1/portals create 0.0.0.0 3260
/iscsi/iqn.2026-04.com.example:target1/tpg1/luns create /backstores/fileio/disk1
/iscsi/iqn.2026-04.com.example:target1/tpg1/acls create iqn.1994-05.com.client:host1
/saveconfig
exit

Expose multiple portals for multipath resilience:

/iscsi/iqn.../tpg1/portals create 10.0.1.10 3260
/iscsi/iqn.../tpg1/portals create 10.0.2.10 3260

Initiator discovery and login:

sudo iscsiadm -m discovery -t sendtargets -p 10.0.1.10
sudo iscsiadm -m node -T iqn.2026-04.com.example:target1 -p 10.0.1.10 -l
sudo iscsiadm -m node -T iqn.2026-04.com.example:target1 -o update -n node.startup -v automatic
lsblk

CHAP example (initiator-side):

sudo iscsiadm -m node -T <TARGET_IQN> -p <TARGET_IP> -o update -n node.session.auth.authmethod -v CHAP
sudo iscsiadm -m node -T <TARGET_IQN> -p <TARGET_IP> -o update -n node.session.auth.username -v CHAPUSER
sudo iscsiadm -m node -T <TARGET_IQN> -p <TARGET_IP> -o update -n node.session.auth.password -v 'CHAP-PASSWORD'

Quick verification and testing

• sudo iscsiadm -m session -P 3 — session details
• sudo multipath -ll — multipath topology
• lsblk and sudo fdisk -l /dev/sdX — device presence
• fio --name=seqwrite --bs=128k --rw=write --size=1G --runtime=60 --filename=/dev/mapper/mpath0 — simple IO test

Troubleshooting commands

• sudo journalctl -u iscsid -f
• sudo dmesg | grep -i iscsi
• sudo iscsiadm -m node -u -T <IQN> -p <IP> — logout
• sudo iscsiadm -m node -o delete -T <IQN> -p <IP> — cleanup node record

Comparison: Target Implementations

Choose LIO for modern in-kernel features and virtualization integration, tgt for small or simple setups, and SCST when vendor or kernel integrations are required.

Common Misconceptions

• iSCSI is not automatically insecure — insecure deployments are the risk. Mitigate with CHAP, IPsec on untrusted networks, VLAN/VRF segmentation, and tight ACLs.
• iSCSI cannot match Fibre Channel — it can match or approach FC latency/throughput for many workloads when the network and NIC offloads are correctly configured (RSS, TSO, iSER/RDMA where available).
• All targets behave the same — feature sets differ (ALUA, vhost, iSER), so test the exact target implementation before production.

Monitoring, Testing and Operational Practices

• Performance tuning: tune queue depths, multipath policies, and interrupt distribution; monitor retransmits and CPU usage during IO tests.
• Monitoring: collect iscsid session stats, NIC counters, multipath state and export via node exporters or custom exporters to Prometheus.
• Backup and snapshots: prefer LVM/ZFS snapshots where supported; ensure snapshots are consistent with application-level flushes (fsfreeze, database dumps).

Related Articles

• Protocol comparison: /iscsi-vs-nfs-vs-smb-protocol-comparison/
• Linux storage primer: /linuxstorage-management-beginners-guide/
• RAID planning: /storageraid-configuration-guide/
• Scale-out alternatives: /ceph-storage-cluster-deployment-beginners-guide/

Production Checklist (short)

• Network: dedicated storage VLAN/VRF, QoS for storage traffic, redundant switches and NICs
• Security: CHAP + restrict ACLs, consider IPsec for multi-tenant or untrusted links
• Redundancy: publish multiple portals across NICs/switches and confirm DM‑Multipath failover
• Testing: run fio workload tests, simulate path failures, validate snapshot/backup workflows
• Observability: collect and alert on session disconnects, retransmits, high latency, and multipath path changes

Appendix: Example multipath.conf and ALUA notes

Example minimal /etc/multipath.conf for many iSCSI arrays:

defaults {
	user_friendly_names yes
	path_selector "round-robin 0"
	rr_min_io 100
}

blacklist {
	devnode "^sd[a-z]$"
}

multipaths {
	multipath {
		wwid    3600a0b8000e1c2d3f4a5b6c7d8e9f0a
		alias   mpath0
	}
}

ALUA (Asymmetric Logical Unit Access) is used by many arrays to mark optimized vs non-optimized paths; ensure your target exposes ALUA states if you rely on path preference. On initiators, verify multipath -ll shows active/optimized paths and test failover by simulating a path down.

References

• IETF RFC 3720 (iSCSI): https://datatracker.ietf.org/doc/html/rfc3720
• LIO / targetcli project: https://linux-iscsi.org/
• open-iscsi (initiator): https://github.com/open-iscsi/open-iscsi