Less DRAM, Same KV Performance: Persistent KV Caching with CXL-Attached AI SSDs

Long-context LLM serving is increasingly limited by KV cache movement, not only by GPU compute. As context length and concurrency grow, KV cache exceeds GPU memory and must be spilled, refilled, prefetched, and reused across the memory hierarchy. This is especially challenging for edge AI servers with 4 to 8 GPUs, where host DRAM capacity, DRAM bandwidth, PCIe bandwidth, power, and cost are tightly constrained. A conventional KV offloading path often uses host DRAM as a staging buffer between GPUs and SSDs. At 4-GPU scale, long-context serving can generate roughly 200 GB/s of KV store and refill traffic. If this traffic must enter and leave host DRAM, the staging path can consume roughly 400 GB/s of DRAM bandwidth, making host DRAM both a capacity requirement and a bandwidth bottleneck.