Inside KV Caching: How AI Models Remember a Conversation Without Rereading Every Word

A language model appears to carry a conversation forward naturally. Underneath, however, it generates one token at a time. Each new token must attend to the tokens that came before it. Recomputing the entire history at every step would repeat an expanding body of work: process the prompt, produce one token, process the prompt plus that token, then begin again.

KV caching is the mechanism that avoids most of this repetition. During attention, the model saves intermediate representations of previous tokens and reuses them when generating the next one. This makes autoregressive inference practical, but creates a different constraint: memory consumption grows with context length, batch size, layer count, and model width.

The result is a revealing infrastructure trade-off. A model can avoid recomputation only by retaining an increasingly large working memory.

What attention needs from every previous token

Inside a transformer attention layer, each token representation is projected into three vectors: a query, a key, and a value. The query expresses what the current token is looking for. Keys describe what earlier tokens offer. Values carry the information that will be combined after the model decides which earlier positions matter.

For a newly generated token, the model creates its query, key, and value at every layer. Its query is compared with the keys of all permitted earlier positions. Those comparisons become attention weights, which determine how the earlier values are combined.

The important asymmetry is that earlier keys and values do not change merely because another token has arrived. They can therefore be stored. On the next decoding step, the model calculates projections only for the new token, appends its key and value to the cache, and attends over the accumulated cache.

This stored collection of keys and values is the KV cache. It is not a transcript, database, or durable memory. It is model-specific numerical state for a particular token sequence and inference session.

Prefill and decoding are different workloads

Serving systems usually divide generation into two phases. During prefill, the model processes the supplied prompt and constructs the initial KV cache. Since the prompt tokens are already available, hardware can process many of their operations in parallel. A long prompt still demands substantial computation, but the workload tends to use accelerators efficiently.

During decoding, the model produces tokens sequentially. Token two depends on token one, so all future steps cannot be calculated together. Each step reads the existing cache, computes the next token’s state, and appends new entries.

Consider a prompt asking a model to compare three contracts, followed by a generated answer. Prefill encodes the contracts and instructions into cached keys and values at every transformer layer. When the model writes the first answer token, its query attends to that stored state. The second token uses the same cache plus the first token’s entries. The cache lengthens until generation stops.

This distinction explains two user-visible delays. Time to first token is strongly influenced by prefill, especially for long prompts. The rate at which subsequent tokens appear is governed more by decoding. Optimizing one phase does not automatically optimize the other.

Why the cache becomes a memory problem

Every cached token needs key and value entries across the model’s relevant attention layers. The storage requirement therefore scales with several dimensions:

• Sequence length: every retained token adds cache entries.
• Concurrent sequences: each active request has its own state.
• Layer count: attention layers maintain separate representations.
• Key and value width: wider representations require more storage.
• Numerical precision: lower-precision formats use fewer bytes per element.

This is why supporting a long context window and serving many simultaneous users are not independent promises. A request occupying more cache memory leaves less accelerator memory for other requests, model weights, temporary tensors, and execution buffers.

The cache also has an awkward lifecycle. It expands token by token, while requests begin and end at different times. A serving engine that reserves one fixed, contiguous region for each sequence may waste capacity or struggle to fit growing requests. Modern systems often divide cache storage into blocks or pages that can be assigned as needed. This resembles virtual memory: logical token positions need not occupy one physically contiguous region.

Paged allocation improves utilization, but it does not remove the underlying demand. It manages fragmentation and scheduling; it does not make retained state free.

Shared prefixes create a second opportunity

Many requests begin identically. A customer-support application may prepend the same policy manual and system instructions to every conversation. Without reuse, each request performs the same prefill computation and stores an equivalent prefix cache.

Prefix caching allows a serving system to retain the KV state for an identical token prefix and attach it to later requests. The saved state must correspond to the same model configuration and exactly the same tokenized prefix. A change in spacing, template structure, model version, or relevant inference settings can invalidate reuse.

Suppose one hundred requests all begin with the same operating handbook but end with different questions. The shared handbook can potentially be prefetched once, while each question and answer creates a private continuation. This can reduce repeated prefill work and, with suitable memory management, avoid storing duplicate physical copies of the common blocks.

Yet the mechanism is narrower than semantic caching. Two paragraphs with the same meaning but different tokens do not normally share KV state. KV reuse depends on computational identity, not conceptual similarity.

How systems shrink or discard the cache

When cache capacity becomes the bottleneck, designers have several options. Each changes a different part of the quality, latency, and throughput equation.

Grouped key-value representations

Some model architectures let multiple query heads share fewer key and value heads. This reduces cache width by design. It can make inference substantially easier to serve, although the model must be trained with that attention arrangement; it is not merely a runtime switch for every existing checkpoint.

Lower-precision storage

Keys and values can be stored in reduced precision. Doing so lowers memory use and bandwidth pressure, but quantization can distort the attention computation. The impact depends on the model, precision format, scaling method, layer, and workload. Evaluation must include long-context cases because small errors can matter differently as the cache grows.

Eviction and sliding windows

A system may retain only recent tokens or selected older positions. Sliding-window attention makes this rule explicit: distant tokens eventually fall outside the active window. Other methods preserve designated anchor tokens or estimate which positions are likely to remain useful.

Eviction introduces a consequential ambiguity. Attention paid in the recent past is not proof that a token will matter later. A forgotten clause, identifier, or instruction may become decisive hundreds of tokens afterward. Cache policies therefore predict future relevance without fully knowing the future query.

Offloading

Cache blocks can be moved from scarce accelerator memory to host memory or other storage tiers. This expands effective capacity but adds transfer latency and consumes interconnect bandwidth. Offloading is most attractive when state would otherwise sit idle; it is less attractive when every decoding step needs it immediately.

What KV caching does not solve

KV caching is sometimes described loosely as model memory, which invites several misconceptions. It does not teach the model new facts. It does not change model weights. It does not decide which conversation details deserve long-term retention. Once the cache is deleted, its state is gone unless the original tokens are supplied and processed again.

It also does not eliminate the cost of long context. Cached keys and values still need to be read during attention, and conventional attention considers an expanding history during decoding. The system has traded repeated projection work for storage and memory traffic.

Nor is a cache generally portable between unrelated models. Its tensors are generated by specific learned projections at specific layers. Even closely related model variants may produce incompatible internal states.

The open questions are now architectural

KV caching began as an inference optimization, but it increasingly shapes model and product design. Several questions remain unsettled.

• Can a system compress old cache entries while preserving exact details that may become relevant much later?
• Should applications structure prompts around reusable prefixes, effectively making cache identity part of prompt architecture?
• Can serving schedulers predict request growth well enough to allocate memory without harming latency?
• When should state remain on an accelerator, move to another tier, or be recomputed?
• Can future attention mechanisms provide durable long-range recall without retaining per-token state at every layer?

The deeper shift is that inference state has become a first-class systems resource. It must be allocated, shared, paged, compressed, evicted, and secured. A long conversation is not merely more text; it is a growing resident object inside the serving stack.

KV caching reveals why the apparent continuity of an AI conversation depends on careful machinery. The model avoids rereading its past by preserving a numerical shadow of it. That shadow makes generation faster, but its expanding footprint sets the boundaries of context, concurrency, and cost.

This post was drafted with AI assistance and reviewed against our editorial policy before publication. Corrections are made at the source, on the page, with the date shown.