Empirical Analysis of KV Cache Offload for Scalable AI Inference Architectures

This talk examines how emerging trends in model architectures and system design such as longer context windows, agent-based workloads, multimodal models, and increasingly heterogeneous infrastructure are reshaping KV Cache design, placement, and storage requirements. As inference scales, system performance becomes increasingly dominated by data movement and memory hierarchy rather than raw compute. We present a detailed analysis of KV Cache I/O paths across modern inference systems and KV Cache management frameworks, highlighting how KV data flows through the system during inference.

These trends expose fundamental challenges at scale, including the lack of interoperability across frameworks and deployments. Differences in KV formats, metadata layouts, and parallelization strategies make it difficult to share and reuse KV Cache across heterogeneous environments. Addressing these challenges requires a system-level approach that rethinks data layout, access patterns, and cross-system compatibility. We conclude by discussing key design considerations and potential directions for enabling efficient and interoperable KV Cache architectures.

Ugur Kaynar

Distinguished Engineer

Dell

Large language model (LLM) inference is rapidly shifting the infrastructure bottleneck from compute throughput to the storage and movement of Key-Value (KV) cache data associated with long-context and high-concurrency inference. While GPUs provide sufficient tensor compute performance for token generation, the scalability and economics of inference are increasingly constrained by the cost, power consumption, and capacity limitations of GPU’s High Bandwidth Memory (HBM). This presentation proposes a new category of infrastructure, purpose-built for AI inference workloads, referred to as an Inference Storage System, that treats transformer KV Cache as a page addressable storage object rather than treating KV Cache as transient GPU memory. The proposed architecture externalizes inactive and shared KV state into a low-latency storage tier accessible through NVMe or RDMA-enabled Ethernet fabrics while preserving active decode performance inside GPU memory.

The presentation examines the architectural requirements for KV-aware storage systems, including paged KV layouts, GPU-direct DMA paths, predictive prefetch, inline KV compression including TurboQuant-class quantization techniques, and attention-aware eviction policies. Unlike traditional block or file storage systems optimized for persistence and throughput, the proposed architecture operates as a KV-page storage appliance capable of storing, indexing, compressing, and streaming transformer KV tensors directly into GPU memory with minimal CPU involvement. The presentation further evaluates the tradeoffs between local NVMe-attached KV storage devices and rack-scale Ethernet-attached KV storage appliances, and discusses how emerging inference runtimes such as vLLM, TensorRT-LLM, and SGLang enable tiered KV-storage hierarchies for scalable inference deployment.

Finally, the presentation argues that KV Cache offload represents the emergence of a new storage systems category optimized specifically for AI inference. In this model, storage infrastructure is no longer designed solely for durability or capacity, but also for sustaining attention locality, maximizing GPU utilization, reducing power consumption, and enabling economically scalable long-context inference. We present architectural considerations, deployment models, and performance implications for future KV storage including both node-local and disaggregated implementations and demonstrate how storage-centric inference architectures may become foundational components of next-generation large-scale AI platforms.

Nader Salessi

Principal

Salessi Technology Advisors

LLM serving systems increasingly rely on KV-cache offloading to extend limited GPU memory, but the storage behavior behind this mechanism is often hidden behind high-level cache-hit metrics. This talk presents a vendor-neutral, block-layer characterization of SSD-backed KV-cache offloading using LMCache and vLLM as a concrete open-source case study.

We compare two practical deployment snapshots: an older LMCache path using buffered I/O and the Linux page cache, and a newer path using prefix-aware deduplication with O_DIRECT. The results show that these two designs do not simply represent “old versus new”; they expose a deeper storage-system trade-off. Buffered I/O can make SSD reads almost disappear in the warm-cache case, but this behavior depends on host memory state outside the KV-cache system’s control. When the page cache is evicted, the same path can suddenly surface as GB-scale SSD reads. In contrast, O_DIRECT makes storage traffic explicit and more predictable, but it can also raise warm-path latency by forcing repeated reads from SSD.

The talk will walk through four stateless LLM workload patterns, block-layer traces, and time-to-first-token measurements. It will show why deduplication provides dramatic write reduction only for exact-repeat workloads, why varied prompt/context traffic can remain write-dominated, and why a KV-cache “hit” is not always a latency win when the retrieved object is too coarse-grained.

Attendees will leave with a practical mental model for AI storage design: when page cache helps, when direct I/O is safer, why KV-cache object granularity matters, and what future SSD-aware LLM serving systems should optimize for.

Dingsen Shi

Phd Student

University of North Texas

Song Fu

Professor of Computer Science and Engineering

University of North Texas

Recent work across the industry has already established that KV Cache reuse across custom shared and disaggregated cache architectures, RDMA‑enabled data paths, and flash‑backed storage can significantly improve the scalability and efficiency of large‑scale LLM inference. These efforts appropriately focus on new systems that elevate the KV Cache to a first‑class resource, often relying on specialized protocols, purpose‑built services, or custom infrastructure.

This presentation takes a different and complementary approach. We share Micron’s measured evidence that capacity‑centric inference designs, combined with targeted upgrades to lower‑cost tiers of fast, persistent storage, can materially improve near‑term performance while strengthening long‑term returns on CapEx. Drawing from a broad, vendor‑agnostic evaluation of KV Cache offload behavior, analyzed from the perspective of NVMe SSDs and grounded in realistic, production‑inspired workloads, we focus on how KV Cache Reads and Writes actually manifest across today’s widely deployed storage, networking, and accelerator environments. Rather than proposing clean‑slate architectures or fabric‑specific solutions, we examine measured access patterns and drive‑level optimizations that naturally align with both sequential and random KV Cache I/O behavior. Attendees will gain actionable tiering heuristics and architectural insights that enable scalable inference by extending existing systems, balancing TCO across the memory‑storage hierarchy, and extracting greater efficiency without disruptive infrastructure changes.

Finally, we extend these insights to architectural considerations for designing dedicated KV Cache tiers or "KV Cache boxes" built from existing ecosystem components such as custom xPUs, interconnects, and high‑performance NVMe storage. Using measured drive behavior as a first‑order input, we discuss how inference systems may evolve as reliance on KV Cache reuse increases with multi‑turn, multi‑agent interactions and shared resources across future‑proof data center infrastructure.

Rohan Mehta

Member of Technical Staff and Systems Performance Engineer

Micron Technology

Long-context LLM serving is increasingly limited by KV cache movement, not only by GPU compute. As context length and concurrency grow, KV cache exceeds GPU memory and must be spilled, refilled, prefetched, and reused across the memory hierarchy. This is especially challenging for edge AI servers with 4 to 8 GPUs, where host DRAM capacity, DRAM bandwidth, PCIe bandwidth, power, and cost are tightly constrained. A conventional KV offloading path often uses host DRAM as a staging buffer between GPUs and SSDs. At 4-GPU scale, long-context serving can generate roughly 200 GB/s of KV store and refill traffic. If this traffic must enter and leave host DRAM, the staging path can consume roughly 400 GB/s of DRAM bandwidth, making host DRAM both a capacity requirement and a bandwidth bottleneck.

Yu-Ming Chang

Engineering Director

Wolley

Disaggregating prefill and decode — running them on separate node pools — is now production reality. Mooncake, SGLang, and vLLM all ship disaggregated backends. The architecture works.

What disaggregation introduces, however, is a set of storage system requirements that are still being worked out. When the KV Cache must cross a network, be shared across decode nodes, survive eviction to lower tiers, be looked up by future requests, and eventually be freed — it behaves as a distributed storage object. The protocols, indexing, consistency models, and garbage collection schemes that managing it demands are active areas of development across both the inference systems and storage communities.

A node holding KV Cache between user turns is occupying accelerator memory for a storage function — keeping state alive — regardless of what compute is happening on that same node. Once you see that, the questions that follow are storage questions: where does the data live, how do you find it, when do you free it. 

Moving KV Cache across nodes — the transfer layer — is where those questions begin. Different protocols make different trade-offs on latency, compatibility, and how efficiently they handle KV Cache that is spread across non-contiguous memory blocks. But transfer alone answers only the first question. The rest belong to the storage system.

The bulk of the talk maps four storage problems that disaggregated inference surfaces: tiering and eviction policy for inference access patterns, distributed prefix indexing with eviction consistency, consistency semantics for immutable content-addressed cache objects, and distributed garbage collection for shared cache blocks with multiple concurrent holders. For each, we look at what exists in the research literature, what is partial, and where the open questions are.

The storage and distributed systems community has prior art on all four. The goal of this talk is to make that connection legible.

Abhishek Gupta

Storage Architect

Cerebras Systems

Rohan Puri

Senior Staff Software Engineer

DataDirect Networks