KV Cache Store
The MASS KV cache store holds KV cache generated during LLM inference and makes it available for reuse across GPU servers. It extends cache capacity beyond a server's GPU and host memory, reducing repeated computation when later requests share the same prompt prefix.
Why Share KV Cache?
During prompt processing, an inference engine computes attention keys and values for the input tokens. It can reuse this KV cache for a matching prefix, such as a shared system prompt or the beginning of a conversation, instead of repeating that computation.
Memory on each GPU server limits how much cache can be retained locally. A shared store provides additional capacity and lets a server retrieve compatible cache produced by another server.
Reuse requires more than matching text. The model, preceding token sequence, and cache representation must be compatible. The inference engine integration identifies which pages can be reused.
Components
| Component | Role |
|---|---|
| Inference engine integration | Identifies cache pages, requests reads, and submits pages for storage |
| Cache service on each GPU server | Maintains cache location information and performs storage I/O |
| Shared NVMe storage | Holds cache pages accessible to the connected GPU servers |
Cache data moves between the GPU server and storage over NVMe-oF. Servers exchange page location updates separately, so the server reading a page does not need to fetch its data from the GPU server that originally produced it.
Saving and Reusing Pages
The store manages KV cache in fixed-size pages. Each page has a key supplied by the inference engine integration and a location in shared storage.
- Save. The cache service writes a completed page to its assigned write area. After the storage write completes, it records the page's location and shares that information with the other servers.
- Look up. When the inference engine requests a page, the cache service checks its local location information. A separate central lookup is not required for each request.
- Read. If the page is available, the cache service reads it from shared storage and returns it to the inference engine, including pages saved by another server.
- Recompute. If the cache cannot supply the page, the inference engine computes the required content. Newly generated pages can be stored for later reuse.
Location updates propagate asynchronously. A newly stored page may therefore be available on one server before another server knows about it.
Managing Cache Space and I/O
Each GPU server writes within its assigned area and can read pages from other servers' areas. This separates write ownership while keeping cached data shared.
As space is reused, the store retires cached pages and waits for reads already using those pages to finish before overwriting them. Pages that have been retired are treated as cache misses on subsequent requests.
The cache service prioritizes reads and limits outstanding writes per device to reduce interference from cache writes. Reads and writes still share device and network resources; latency depends on the workload and available capacity.
Management and Storage Behavior
Administrators create stores, select the GPU servers allowed to connect, and monitor usage in the Dashboard. A GPU server connects with mass-ctl kvcache connect <store>. Its inference engine integration handles cache reuse during inference.
KV cache is regenerable data. Eviction or an unavailable page can increase computation and response time, but the engine can rebuild the missing cache. File & Object Storage serves a different purpose: retaining application data under a selected protection policy.