Compressing the LLM Context Window

Large language model context windows have grown from a few thousand tokens to several million in three years, and the cost of feeding them has grown with them. A single long-running coding agent or enterprise workflow can ship tens of billions of tokens per month through an inference provider, with the majority of those tokens being repeated boilerplate, verbose tool outputs, and stale conversation history. The industry's default answer — bigger context windows and longer prompts — ignores a body of work the storage community settled decades ago: when reads are expensive, you compress, you cache, and you dereference.

This talk treats the LLM prompt as a storage object and walks through the compression pipeline required to operate it efficiently at scale. We will cover three layers.

First, prefix-aligned caching. Provider-side prompt caches (Anthropic, OpenAI, Google) behave like log-structured stores with strict prefix boundaries: a single byte of drift in the early portion of a prompt invalidates the entire suffix. We will examine the failure modes — timestamps, session IDs, dynamic system prompts, retrieved snippets injected in the wrong order — and the alignment techniques that extract every available cache hit before any compression happens.

Second, content-aware semantic compression. Naive truncation throws away the wrong tokens. Modern approaches use small classifier models — we focus on a ModernBERT-class dual-head architecture, roughly 150MB quantized, sub-5ms inference on commodity hardware — that score tokens for keep/discard against a task-conditioned objective. We will walk through training data construction (LLM-distilled labels rather than meeting-transcript corpora), the dual-head design separating token-level decisions from span-level importance, and the measured tradeoff curves between compression ratio and downstream task fidelity.

Third, content-addressed retrieval as a hot/cold tier. Aggressive compression destroys information the model may later need. The solution is familiar: keep a content-addressed store of original payloads keyed by hash, inject a retrieval tool into the model's function-call surface, and let the model dereference on demand. This is, structurally, the same pattern as a deduplicated object store with on-read rehydration, applied to a new tier.

We share production numbers from a wire-level open-source proxy deployed in front of coding agents and enterprise workflows: 40–90% token reduction depending on content class, the cache-hit-rate gains attributable to alignment alone, and the storage and latency characteristics of the content-addressed tier at multi-billion-token scale.

Attendees will leave with a working mental model of the LLM context window as a tiered, compressible, content-addressed storage object, and with concrete design patterns drawn from existing compression and caching literature that apply directly to it.