Mooncake

Overview

Mooncake is an infrastructure project for large-scale LLM inference and training. Its KV-cache-centric, disaggregated architecture separates compute roles such as prefill and decode, while Mooncake Store provides a distributed object layer for reusable KV caches and model weights. This lets serving frameworks share cached state across instances, reduce repeated prefill work, and extend cache capacity beyond a single GPU or host.

Prerequisites

• Install the same MASS client and DAOS-enabled Mooncake build on every participating host.
• Mount the same MASS volume on every host and verify that DFS_PATH resolves to its POSIX container.
• For a multi-host run, configure passwordless SSH and passwordless sudo for the launch user on every host. Alternatively, set RSH_USER=root and configure passwordless root SSH.

note

For Storage resource provisioning, see Managing Storage. For mass-client installation instructions, see CLI Reference.

MASS-Mooncake Integration Guide

Prerequisites

Mooncake's DAOS backend requires the full mass-client package on the host where Mooncake is built and run. The package provides the DAOS client libraries, headers, and mass.pc pkg-config module used by CMake.

Install mass-client

Follow the mass-client installation guide to configure the MangoBoost package repository and install the client. Use a full client package (Ubuntu or RHEL/Rocky Linux 9); the RHEL/Rocky Linux 10 package is a thin client and does not include the DAOS development files required here.

Prepare the Mooncake source tree and install its common build dependencies:

cd /path/to/Mooncake
sudo bash dependencies.sh

Apply the Mooncake integration patch to the Mooncake source tree before configuring the build:

MOONCAKE_SRC=/path/to/Mooncake
PATCH_FILE=/path/to/downloaded/mooncake.patch

cd "$MOONCAKE_SRC"
git apply --check "$PATCH_FILE"
git apply "$PATCH_FILE"

After the patch is applied, mass-client's pkg-config metadata resolves the DAOS include and library paths automatically. No separate PKG_CONFIG_PATH or DAOS_ROOT setting is required:

pkg-config --exists mass
pkg-config mass --cflags --libs

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DWITH_STORE=ON \
  -DUSE_DAOS=ON

cmake --build build -j"$(nproc)"
sudo cmake --install build

Configure Mooncake to use MASS

Create or select a MASS volume for Mooncake and mount the same POSIX/UNS path on every Mooncake host. The path must resolve to the DAOS pool and container (for example, /mnt/mass/mooncake); the adapter uses it for DAOS name resolution and sends object I/O through libdfs.

Start the Mooncake master with offload enabled:

mooncake_master --enable_offload=true

Set the distributed backend variables on every Mooncake client before starting it:

export MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=distributed
export MOONCAKE_DISTRIBUTED_FS_TYPE=daos
export MOONCAKE_DISTRIBUTED_ROOT_DIR=/mnt/mass/mooncake

Start each client with offload enabled as well. With this configuration, Mooncake Store persists offloaded objects in MASS through the DAOS adapter. Set MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR to distributed only for clients that should use the MASS backend; other clients can continue using the default local backend.

Start a Mooncake client before running the benchmark. Adjust the host address, MASS path, and offload directory for your environment:

MOONCAKE_DISTRIBUTED_FS_TYPE=daos \
  MOONCAKE_DISTRIBUTED_ROOT_DIR=/mnt/poc-home/vllm-kv \
  MOONCAKE_DISTRIBUTED_HEALTH_CHECK=true \
  MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/var/tmp/mooncake-offload-stub \
  MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS=2 \
  UCX_IB_RCACHE_MAX_REGIONS=256 UCX_RCACHE_MAX_REGIONS=256 \
  /opt/Mooncake/bin/mooncake_client \
    --master_server_address=127.0.0.1:50051 \
    --metadata_server=P2PHANDSHAKE \
    --host=211.250.100.47 --port=50052 \
    --protocol=rdma \
    --global_segment_size="32 GB" \
    --local_buffer_size="1 GB" \
    --enable_offload=true

Benchmark

DAOS Adapter

daos_adapter_bench.sh is an IOR-like benchmark for measuring the read and write performance of Mooncake's DAOS adapter on MASS.

The examples below assume Mooncake is installed under /opt/Mooncake. Adjust the prefix if it is installed elsewhere.

Running the Benchmark

MPI rank

NPROCS is the number of MPI ranks started on each host. The total rank count is NPROCS multiplied by the number of entries in HOSTS. Each rank creates NR objects, so the total dataset size for one write pass is number of hosts × NPROCS × NR × SIZE.

Run a single-host test and write the machine-readable result on rank 0:

NPROCS=4 SUMMARY=/tmp/daos-adapter.json DFS_PATH=/mnt/mooncake-vol \
  /opt/Mooncake/scripts/daos_adapter_bench.sh

To measure aggregate throughput from multiple clients, provide a comma-separated host list. In this example, four ranks run on each host and eight ranks run in total:

HOSTS=client01,client02 NPROCS=4 \
  SUMMARY=/tmp/daos-adapter.json DFS_PATH=/mnt/mooncake-vol \
  /opt/Mooncake/scripts/daos_adapter_bench.sh

The script re-executes itself with sudo -E when necessary. It prefers MPICH when available and otherwise uses Open MPI. Before a multi-host workload starts, it checks remote access, the benchmark binary, and the full MPI launch path.

Configuration

NPROCS and SUMMARY are mandatory. All other variables are optional.

Variable	Description	Default
NPROCS	MPI ranks to run on each host.	None
SUMMARY	JSON result path written by rank 0. Its parent directory must exist on the rank-0 host.	None
HOSTS	Comma-separated hosts. Every host must have the same installation and MASS target.	localhost
DFS_PATH	MASS dfuse/UNS path used to resolve the target pool and container. Data I/O uses libdfs.	/mnt/mooncake-vol
DIR	Benchmark subdirectory under DFS_PATH.	adapterbench
NR	Objects created by each rank. Increasing ranks while retaining this value produces a weak-scaling workload.	512
SIZE	Size of each object; accepts bytes or a k, m, or g suffix.	16m
XFER	Maximum size of each dfs_write; 0 leaves writes unsplit.	0
READ_XFER	Maximum size of each dfs_read; 0 leaves reads unsplit.	8m
CHUNK	DFS chunk size used when creating files.	8m
THREADS	Concurrent I/O threads in each rank.	8
READ_ROUNDS	Number of complete read passes over the dataset.	1
WR_ONLY	Set to 1 to run only writes and retain the files for a later read-only run.	0
RD_ONLY	Set to 1 to read files retained by a matching write-only run.	0
CLEANUP	Set to 1 to delete benchmark files after the run. A write-only run defaults to retaining them.	1, or 0 with WR_ONLY=1
RSH_USER	SSH login used for multi-host launch. It needs passwordless SSH and passwordless sudo; use root for root-to-root SSH.	Invoking user, or root when already root
DAOS_AGENT_DRPC_DIR	Directory containing daos_agent.sock.	/run/boostx-agent, or an active /tmp/boostx-agent-* directory
MPIRUN	MPI launcher override.	mpirun.mpich when available, otherwise mpirun
EXTRA_ARGS	Additional flags passed verbatim to daos_adapter_bench, such as --align=0 --iovcnt=4.	Unset
DRYRUN	Set to 1 to print the generated command without running it.	0

For a separate write and read test, retain the write dataset and then use exactly the same target, host list, rank count, object count, and object size:

HOSTS=client01,client02 NPROCS=4 NR=128 SIZE=1g WR_ONLY=1 \
  SUMMARY=/tmp/daos-write.json DFS_PATH=/mnt/mooncake-vol \
  /opt/Mooncake/scripts/daos_adapter_bench.sh

HOSTS=client01,client02 NPROCS=4 NR=128 SIZE=1g RD_ONLY=1 \
  SUMMARY=/tmp/daos-read.json DFS_PATH=/mnt/mooncake-vol \
  /opt/Mooncake/scripts/daos_adapter_bench.sh

Interpreting Results

The console prints one line per rank and a [AGG] line. The aggregate bandwidth divides the total successful bytes by the slowest rank's wall time, which prevents a fast rank from hiding an imbalanced client. The JSON summary records the effective configuration and, for every write or read round, failed_ops, wall time, MiB/s, objects/s, p50/p95/p99/max object latency, and per-rank throughput.

The workload controls intentionally resemble IOR: SIZE is comparable to IOR's block size, while XFER and READ_XFER are comparable to its transfer size. IOR defines a block as the contiguous data handled by one client and a transfer as the buffer submitted by one I/O call; see the IOR options and the MASS IOR guide. Use matching process counts and data volumes when comparing results, but expect some difference because this benchmark creates Mooncake-style objects and calls DaosAdapter directly.

This microbenchmark does not replace IO500. IO500 combines IOR bandwidth phases with mdtest and parallel find phases to cover both data and metadata behavior. Use the MASS IO500 guide for system-level acceptance or comparison testing. For an official IO500 submission, all mandatory phases and the 300-second write-phase stonewall requirement still apply; see the IO500 run documentation and submission rules.

Storage KV

store_kv_bench.py measures the end-to-end Mooncake Store path. Unlike the DAOS Adapter microbenchmark, it includes Store allocation and metadata operations, object placement, Transfer Engine traffic, and the selected put or get API. Use it to validate KV operations and measure the throughput and latency seen by a Mooncake application.

The script is located at:

/opt/Mooncake/mooncake-store/benchmarks/store_kv_bench.py

Adjust the prefix if Mooncake is installed elsewhere. The filename is store_kv_bench.py, not storage_kv_bench.py. Run python3 store_kv_bench.py --help to check the options supported by your installed Mooncake version.

Prerequisites

• Complete the MASS-Mooncake integration earlier on this page, including the full mass-client package and a DAOS-enabled Mooncake build.
• Mount the same MASS volume on every participating Mooncake host.
• Start a Mooncake master that has offload and a metadata service enabled.
• Use a unique, reachable --local-hostname endpoint for every benchmark process. RDMA devices are discovered automatically.

For example, configure the MASS backend and start a master with its embedded HTTP metadata service:

export MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=distributed
export MOONCAKE_DISTRIBUTED_FS_TYPE=daos
export MOONCAKE_DISTRIBUTED_ROOT_DIR=/mnt/mass/mooncake

mooncake_master \
  --enable_offload=true \
  --offload-backend distributed \
  --distributed-fs-type daos \
  --distributed-root-dir /mnt/mass/mooncake \
  --enable_http_metadata_server=true \
  --http_metadata_server_host=0.0.0.0 \
  --http_metadata_server_port=8080

Keep the backend variables set in the shell that starts each benchmark process.

Validate the Setup

Start with a small write-and-read verification run. Replace the example addresses with values reachable from the client:

python3 /opt/Mooncake/mooncake-store/benchmarks/store_kv_bench.py \
  --scenario verify_write \
  --local-hostname 10.0.0.21:50071 \
  --metadata-server http://10.0.0.10:8080/metadata \
  --master-server 10.0.0.10:50051 \
  --protocol rdma \
  --global-segment-size 1073741824 \
  --local-buffer-size 268435456 \
  --nr-objects 64 \
  --value-size 1048576 \
  --pattern 0xA5 \
  --verify \
  --summary-json /tmp/store-kv-verify.json

For a TCP-only validation, use --protocol tcp and addresses reachable over the TCP network.

verify_write writes every object and reads it back. The run succeeds when the process exits with status 0, the console reports no failed KVs or verification failures, and the summary JSON has "ok": true.

Run Throughput Workloads

The following example writes 512 objects of 16 MiB each with eight concurrent jobs. It contributes a 12 GiB memory segment to Mooncake Store and uses a 2 GiB local Transfer Engine buffer:

python3 /opt/Mooncake/mooncake-store/benchmarks/store_kv_bench.py \
  --scenario write_perf \
  --local-hostname 10.0.0.21:50071 \
  --metadata-server http://10.0.0.10:8080/metadata \
  --master-server 10.0.0.10:50051 \
  --protocol rdma \
  --global-segment-size 12884901888 \
  --local-buffer-size 2147483648 \
  --io-api plain \
  --numjobs 8 \
  --iodepth 1 \
  --batch-size 8 \
  --nr-objects 512 \
  --value-size 16777216 \
  --pattern 0xA5 \
  --summary-json /tmp/store-kv-write.json

To measure reads, change the scenario to read_perf and add --verify. By default, read_perf first creates the dataset in a prepare_write phase and then reads it in a separate read_perf phase:

python3 /opt/Mooncake/mooncake-store/benchmarks/store_kv_bench.py \
  --scenario read_perf \
  --local-hostname 10.0.0.21:50071 \
  --metadata-server http://10.0.0.10:8080/metadata \
  --master-server 10.0.0.10:50051 \
  --protocol rdma \
  --global-segment-size 12884901888 \
  --local-buffer-size 2147483648 \
  --numjobs 8 --iodepth 1 --batch-size 8 \
  --nr-objects 512 --value-size 16777216 \
  --pattern 0xA5 --verify \
  --summary-json /tmp/store-kv-read.json

For a time-based mixed workload, use mixed_rw, set a positive runtime, and select the read percentage. --prepare-objects controls the initial readable dataset and --write-objects limits the object IDs available for writes:

python3 /opt/Mooncake/mooncake-store/benchmarks/store_kv_bench.py \
  --scenario mixed_rw --runtime 60 --rwmixread 70 \
  --prepare-objects 512 --write-objects 4096 \
  --local-hostname 10.0.0.21:50071 \
  --metadata-server http://10.0.0.10:8080/metadata \
  --master-server 10.0.0.10:50051 \
  --protocol rdma \
  --global-segment-size 12884901888 \
  --local-buffer-size 2147483648 \
  --numjobs 8 --iodepth 1 --batch-size 8 \
  --nr-objects 512 --value-size 16777216 \
  --pattern 0xA5 --verify \
  --summary-json /tmp/store-kv-mixed.json

Configuration

Option	Description	Default
--scenario	Workload to run: validation, fill, read/write performance, mixed I/O, metadata, removal, or replay.	Required
--local-hostname	Reachable address for this Store client. Each simultaneous process needs a unique endpoint.	127.0.0.1:50071
--metadata-server	Transfer Engine metadata service URL or P2PHANDSHAKE.	http://127.0.0.1:8080/metadata
--master-server	Mooncake master address.	127.0.0.1:50051
--protocol, --device-name	Transfer protocol. RDMA devices are discovered automatically; use --device-name only to override device selection.	tcp, empty
--global-segment-size	Bytes of memory contributed by this process to the global Store pool.	64 MiB
--local-buffer-size	Bytes reserved for the local Transfer Engine buffer.	32 MiB
--io-api	plain uses the regular Store API; zcopy uses registered buffers.	plain
--numjobs, --iodepth	Concurrency controls. The number of worker lanes is numjobs × iodepth.	1, 1
--batch-size	KV objects submitted by one batch request.	1
--nr-objects	Object count for an object-count-based phase.	128
--runtime	Runtime in seconds; 0 selects an object-count-based run. Required by mixed_rw.	0
--value-size	Value size in bytes. Write workloads require a multiple of 512 bytes.	4 KiB
--pattern, --verify	Reuse a fixed payload and optionally verify data read back. --verify requires --pattern.	Empty, disabled
--summary-json	Machine-readable phase and overall results.	Unset

Use --pattern for throughput comparisons. Without it, payload generation can become part of the timed path and limit Python-side throughput. When using --io-api zcopy, ensure that value-size × batch-size × numjobs × iodepth does not exceed --local-buffer-size, and first confirm that the installed Mooncake build supports registered buffer allocation.

The public script runs one Store client process and does not launch ranks or aggregate results across hosts. For a multi-host test, start one or more processes on each host, assign every process a unique --local-hostname address or port, and prevent key collisions with distinct --object-id-start ranges or --key-prefix values. Save a separate summary file for each process and aggregate phase results after all processes finish.

Interpreting Results

The console and summary JSON report request and KV counts, successful bytes, duration, requests/s, KVs/s, MiB/s, p50/p95/p99 latency, misses, verification failures, and error counts. For read_perf and mixed_rw, compare the named main phase rather than the overall value, because overall also includes the dataset preparation phase.

Store throughput and MASS persistence throughput are different measurements. A successful Store put can return before asynchronous offload to MASS has drained. Use Mooncake's offload or master metrics to confirm that the queue is empty and measure persistence separately. Avoid recursively listing or running stat on every generated object merely to monitor progress; a large namespace scan can distort the workload and place unnecessary load on the mounted filesystem. +

Notes

• Benchmark the full Mooncake Store path separately. DAOS adapter throughput excludes scheduling, RPC, hashing, offload queues, and cache hit/miss behavior.
• A read-only run requires the files from a matching write-only run. Do not change DFS_PATH, DIR, HOSTS, NPROCS, NR, or SIZE between the two phases.
• Record the MASS client and Mooncake versions, participating hosts, workload variables, and client topology with every result so later comparisons remain meaningful.