Storage Engine

Server → Engine → Target Hierarchy

The storage system is organized in a three-level hierarchy: server → engine → target.

Storage Engine — Independent Unit of Work Within a Server

A single storage server runs one or more storage engines. Each engine operates as an independent process with its own network endpoint, memory space, and I/O queues.

Why multiple engines per server: NUMA isolation

Modern servers have multiple CPU sockets, each with its own local DRAM bank (NUMA topology). Accessing memory on another socket incurs tens of nanoseconds of additional latency. Binding an engine to a single NUMA node ensures all memory accesses for that engine are served from local DRAM, making cross-NUMA latency zero.

As a result, adding more engines within a server scales throughput linearly with the number of NUMA nodes, and a failure in one engine does not interrupt I/O in other engines.

I/O Target — Smallest I/O Unit Dedicated to One CPU Core

Each engine contains multiple I/O targets. A target is the smallest independent execution unit for processing data reads and writes.

CPU cores and targets are always bound 1:1. Each target is pinned to a dedicated CPU core, and configurations where one core serves multiple targets or multiple cores serve one target are not supported. This design eliminates scheduling contention between targets and enables throughput to scale in direct proportion to core count.

Each target has its own independent SPDK I/O queue, so there is no lock contention between targets.

Target and NVMe Mapping

The default configuration assigns one target to one NVMe device. When a server has more CPU cores than NVMe devices, a single physical NVMe can be partitioned into multiple namespaces, shared by multiple targets. In this case, each target still maintains its own independent I/O queue with no contention.

Configuration	Target : CPU Core	Target : NVMe
Standard (recommended)	1 : 1	1 : 1
Core surplus	1 : 1	N : 1 (namespace partitioning)

Metadata Storage

Each I/O target has two storage areas:

• DRAM: Acts as a write buffer. Immediately receives and acknowledges (ACK) write requests and asynchronously flushes to NVMe.
• NVMe: Permanently stores the metadata index. Recoverable from NVMe after restart or failure.

This structure allows metadata I/O to be processed at DRAM speed while NVMe ensures durability.

Cluster Management Service

Three nodes in the cluster form the cluster management service using Raft consensus.

Cluster management service roles:

• Pool Map maintenance: Records the configuration and state of all servers, engines, and targets in the cluster.
• Fault detection and propagation: When a node failure is detected, propagates the updated Pool Map to all clients.
• Off the data path: Only notifies about Pool Map changes and does not participate in actual I/O.

Even during leader re-election, I/O from already-connected clients continues uninterrupted.

Data Placement Algorithm

Clients directly compute which target on which server to store data by combining the Object ID with the Pool Map.

• No metadata server lookup: No central server is queried for each placement decision, making this overhead zero.
• Deterministic: The same Object ID always maps to the same target, eliminating the need for re-lookup.
• Client-side distributed computation: Each client computes independently, so there is no bottleneck at the placement decision stage.

When the Pool Map changes (node failure, node addition), the cluster management service notifies clients, and clients recompute using the new Pool Map.

Replication / Erasure Coding

A protection policy is selected at VolumeGroup creation time. Replicas and parity blocks are always placed on targets on different servers.

Policy	Configuration	Protection Level	Capacity Overhead
Replica 2	2 copies	Protects against 1 server failure	2×
Replica 3	3 copies	Protects against 2 simultaneous server failures	3×
EC 2+1 (Standard)	2 data + 1 parity	Protects against 1 server failure	1.5×
EC 4+2 (Enterprise)	4 data + 2 parity	Protects against 2 simultaneous server failures	1.5×
EC 6+3 (High Redundancy)	6 data + 3 parity	Protects against 3 simultaneous server failures	1.5×

Replica and EC are mutually exclusive. For large-scale workloads, EC provides better capacity efficiency for the same level of protection.