Enabling Ultra-High-Scale RAG with Cost-Efficient, All-in-Storage ANNS with High-Capacity SSDs
Winchester
Tue Sep 29 | 2:35pm
Abstract
As semantic search and Retrieval-Augmented Generation (RAG) systems scale to billions or even trillions of vectors, the traditional DRAM-based vector search solutions become prohibitively expensive and challenging to scale. The demand for high-scale RAG continues to grow as enterprises seek to index, search, and reason over ever-larger volumes of information.
This talk will explore scaling cost, index build time, and high-capacity SSD-based architectures to efficiently serve large scale semantic search workloads. We will demonstrate that high-density SSD systems can meet RAG latency and throughput requirements at the scale of 10s/100s of billions of vectors offering significantly lower cost, scalable, and a sustainable alternative to memory-based approaches. This session will share the architectural insights of an SSD-friendly vector search engine technology, performance results, and highlight the SSDs as a practical foundation for next generation, web scale semantic search RAG deployments.
Related Sessions
Data aware Open Context Engine for AI Agents
Enterprises are deploying autonomous AI agents at scale. These agents connect to operational systems through the Model Context Protocol (MCP), an open standard now governed by the Linux Foundation with various types of data sources. However, this does not ensure the "real understanding" of the data to be inferred by the AI Agents accurately. SODA Contexture is an open source project from SODA Foundation (a sub-foundation under LF) that focuses on building actionable context from raw operational data. It brings OCS (Open Context Specification) to define the data-aware-context for various data sources and types. It provides a method of providing dependencies and relations. OCS based connectors for different data sources ensure improved context and enhanced accuracy in the inference and decision-making by AI Agents and LLM Models.
The session will be driven by a demo where we will showcase the use case, preparation of OCS, building data-aware-context and how it provides intuitive inferences.
We will also share the next enhancements, and it will provide the community members with an opportunity to collaborate on SODA Contexture project.
Enabling Ultra-High-Scale RAG with Cost-Efficient, All-in-Storage ANNS with High-Capacity SSDs
As semantic search and Retrieval-Augmented Generation (RAG) systems scale to billions or even trillions of vectors, the traditional DRAM-based vector search solutions become prohibitively expensive and challenging to scale. The demand for high-scale RAG continues to grow as enterprises seek to index, search, and reason over ever-larger volumes of information.
This talk will explore scaling cost, index build time, and high-capacity SSD-based architectures to efficiently serve large scale semantic search workloads. We will demonstrate that high-density SSD systems can meet RAG latency and throughput requirements at the scale of 10s/100s of billions of vectors offering significantly lower cost, scalable, and a sustainable alternative to memory-based approaches. This session will share the architectural insights of an SSD-friendly vector search engine technology, performance results, and highlight the SSDs as a practical foundation for next generation, web scale semantic search RAG deployments.
PageANN: Scalable I/O-Efficient Disk-Based Approximate Nearest Neighbor Search with Disk Page-Aligned Graph
Approximate Nearest Neighbor Search (ANNS) is a critical component of modern AI/ML systems, particularly in retrieval-augmented generation (RAG). While graph-based ANNS methods provide high query efficiency, their scalability is limited by host memory. Disk-based approaches mitigate this by offloading data to SSDs, but they suffer from a key bottleneck: as datasets grow and memory shrinks, they incur high I/O latency or fail due to insufficient memory.
We present PageANN, a scalable and I/O-efficient disk-based ANNS framework that maintains high performance across varying memory budgets. PageANN aligns graph node granularity with SSD page granularity, treating each page as a search unit. By grouping nearby vectors within the same page and constructing shared neighbor lists, each I/O operation retrieves only relevant data, expanding the effective search scope while reducing redundant accesses. PageANN further incorporates a lightweight in-memory hash-based routing index to reduce search hops and improve navigation efficiency. By packing all necessary information for each search step within a single page, it enables on-demand data fetching with low memory overhead, while still benefiting from additional memory when available.
Experiments show that PageANN outperforms state-of-the-art disk-based ANNS methods, achieving up to 1.94× higher throughput and 66.8% lower latency across datasets and memory settings, without sacrificing recall accuracy.
Compressing the LLM Context Window
Large language model context windows have grown from a few thousand tokens to several million in three years, and the cost of feeding them has grown with them. A single long-running coding agent or enterprise workflow can ship tens of billions of tokens per month through an inference provider, with the majority of those tokens being repeated boilerplate, verbose tool outputs, and stale conversation history. The industry's default answer — bigger context windows and longer prompts — ignores a body of work the storage community settled decades ago: when reads are expensive, you compress, you cache, and you dereference.
This talk treats the LLM prompt as a storage object and walks through the compression pipeline required to operate it efficiently at scale. We will cover three layers.
First, prefix-aligned caching. Provider-side prompt caches (Anthropic, OpenAI, Google) behave like log-structured stores with strict prefix boundaries: a single byte of drift in the early portion of a prompt invalidates the entire suffix. We will examine the failure modes — timestamps, session IDs, dynamic system prompts, retrieved snippets injected in the wrong order — and the alignment techniques that extract every available cache hit before any compression happens.
Second, content-aware semantic compression. Naive truncation throws away the wrong tokens. Modern approaches use small classifier models — we focus on a ModernBERT-class dual-head architecture, roughly 150MB quantized, sub-5ms inference on commodity hardware — that score tokens for keep/discard against a task-conditioned objective. We will walk through training data construction (LLM-distilled labels rather than meeting-transcript corpora), the dual-head design separating token-level decisions from span-level importance, and the measured tradeoff curves between compression ratio and downstream task fidelity.
Third, content-addressed retrieval as a hot/cold tier. Aggressive compression destroys information the model may later need. The solution is familiar: keep a content-addressed store of original payloads keyed by hash, inject a retrieval tool into the model's function-call surface, and let the model dereference on demand. This is, structurally, the same pattern as a deduplicated object store with on-read rehydration, applied to a new tier.
We share production numbers from a wire-level open-source proxy deployed in front of coding agents and enterprise workflows: 40–90% token reduction depending on content class, the cache-hit-rate gains attributable to alignment alone, and the storage and latency characteristics of the content-addressed tier at multi-billion-token scale.
Attendees will leave with a working mental model of the LLM context window as a tiered, compressible, content-addressed storage object, and with concrete design patterns drawn from existing compression and caching literature that apply directly to it.
LPDDR Memory: A New CPU Memory Choice for AI Inference
As Large Language Models (LLM) evolve and become integrated across various networks, there is a growing demand for more advanced AI - Training and inference hardware to support their capabilities. The compute power for these LLMs is becoming increasingly constrained by memory bandwidth and power. AI Inference is now the dominant workload in modern computing, demanding new architectural approaches to handle data movement, retention, and processing.
LPDDR SDRAM, originally developed for low-power mobile products, is emerging as an alternative CPU memory solution for AI inference workloads. LPDDR offers a good balance of power efficiency, performance, capacity, and cost. LPDDR5X, now deployed in AI data centers in SOCAMM2 form factor, has demonstrated improved system performance with lower power on many of the latest LLMs. This presentation will look at implementation options, performance benchmarks, and the evolution to LPDDR6 for next generation systems.
Storage for AI 104 - A Continuing Intro to Storage for Inference
Building on Storage for AI 103, which introduced why Inference needs KV Cache (context memory storage) and storage for KV Cache, this presentation will go deeper into the key types of storage used in Inference and how the workloads of KV Cache and RAG with vector db search or RAG with graph neural network search leads to three different types of workloads with different requirements for the backing storage.
