Inside KV Caching: The Memory System That Makes Long AI Conversations Possible

A conversational AI interface creates a persuasive illusion: the model appears to hold an exchange in mind, then continue from where it left off. Under the surface, however, a transformer does not possess a persistent stream of thought. It receives tokens and performs numerical operations over them. Without a particular inference optimization, generating every new token would require repeatedly recomputing much of the conversation.

That optimization is the key-value cache, usually shortened to KV cache. It retains intermediate attention data from earlier tokens so the model can reuse it during generation. The mechanism is fundamental to responsive chat, but it also consumes substantial accelerator memory. As context windows and concurrent workloads grow, that memory burden becomes one of the defining constraints of AI serving.

The computation a transformer would otherwise repeat

Text first becomes a sequence of token representations. Within each transformer attention layer, the model projects every representation into three vectors: a query, a key, and a value. The query expresses what the current position is looking for. Keys describe what previous positions offer. Values contain the information that can be retrieved.

Attention compares a query with available keys, converts those comparisons into weights, and uses the weights to combine corresponding values. In causal language generation, a token can attend only to itself and earlier positions.

Consider the prompt The observatory opened at midnight. To predict what follows, the model computes keys and values for the prompt tokens. Suppose it generates because. On the next step, it must predict a token using the original prompt plus because. The keys and values for the original tokens have not changed. Reconstructing them would duplicate work.

The KV cache stores those vectors at every attention layer. On the next decoding step, the model computes a new query, key, and value only for the latest token. It appends the new key and value to the cache, then compares the query against the accumulated keys. Generation remains sequential, but much of the static prefix computation disappears.

Prefill and decode are different workloads

Inference is commonly divided into two phases whose resource profiles differ sharply.

During prefill, many prompt positions can be processed in parallel. A long document therefore creates considerable initial work, but accelerators can apply their computational capacity across many tokens at once. During decode, each output token depends on the preceding output token. This dependency limits parallelism within one sequence.

Decode repeatedly reads keys and values accumulated across layers and positions. Moving this data through memory can become more important than arithmetic throughput. Consequently, an accelerator capable of enormous numerical performance may still generate slowly if memory bandwidth or cache capacity is the tighter constraint.

This distinction also explains why one latency number is inadequate. A system may begin answering quickly yet stream slowly, or spend longer ingesting a large prompt and then decode smoothly. Time to first token and inter-token latency describe different parts of the path.

Why the cache becomes large

The cache grows with several model and workload dimensions:

• Sequence length: every retained token contributes keys and values.
• Layer count: each attention layer maintains its own cache.
• KV head count: more independent key-value heads require more stored vectors.
• Head dimension: wider vectors occupy more memory.
• Numerical precision: each element may use a larger or smaller representation.
• Concurrent sequences: each active request needs cache space unless some prefix data can be shared.

A useful conceptual formula is:

KV memory per sequence is proportional to tokens × layers × KV heads × head dimension × two vectors × bytes per element.

The factor of two represents keys and values. Queries need not persist after the current attention operation, so they are not cached in the same way.

This scaling creates an architectural tension. A long context window may be supported by the model, yet serving many long conversations simultaneously can exhaust available memory. Maximum context length is therefore not merely a model capability. It is also a capacity-planning decision involving concurrency, latency targets, and hardware.

The techniques that bend the constraint

Grouped and multi-query attention

Standard multi-head attention can assign separate key and value projections to each query head. Multi-query attention lets many query heads share one set of keys and values. Grouped-query attention occupies a middle ground: groups of query heads share KV heads.

Reducing KV head count shrinks the cache and the data that decode must read. The trade-off is model quality and behavior: sharing representations may reduce expressive flexibility, and the attention architecture is generally determined during model design or adaptation rather than switched freely at serving time.

Cache quantization

Keys and values can be stored at lower precision. Smaller elements reduce capacity and bandwidth requirements, but quantization introduces approximation. Its impact can vary by layer, token, model, and task. An implementation must also account for the cost of quantizing, dequantizing, and managing scaling metadata.

The practical question is not whether compressed values differ from their originals—they do—but whether those differences meaningfully alter output quality while improving serving efficiency.

Paged allocation

Requests arrive with different prompt lengths, generate different numbers of tokens, and finish unpredictably. Reserving one contiguous maximum-size cache region per request wastes memory and causes fragmentation.

Paged KV systems divide cache memory into blocks and map logical token positions to physical blocks. A sequence can acquire blocks as it grows and release them when it ends. This resembles virtual-memory ideas in operating systems. Indirection introduces management overhead, but utilization usually improves under irregular workloads.

Prefix caching

Many requests begin with identical tokens: a system instruction, a document template, or a stable few-shot prefix. Because deterministic model computations over an identical prefix produce reusable KV entries, a server can retain and share those entries instead of repeating prefill.

Reuse requires exactness at the token and model-configuration level. A small textual change may alter tokenization and break the match. Position handling, adapter selection, and cache isolation must also be compatible. Prefix caching reuses computation; it does not mean semantically similar prompts can safely share arbitrary KV states.

A worked serving example

Imagine an assistant whose requests contain three segments: a fixed policy prompt, a changing project brief, and the live conversation. The fixed policy occupies the beginning of every request.

1. The first request prefills all three segments and creates KV entries for each token.
2. The server records the block boundary covering the fixed policy prefix.
3. A later request with the exact same policy references those cached blocks rather than recomputing them.
4. New blocks are allocated for its project brief and conversation.
5. As the model generates, each output token adds KV data across all layers.
6. When the request ends, private blocks are released; shared policy blocks may remain while useful and permitted.

This arrangement lowers repeated prefill work, but it does not eliminate decode growth. A prolonged conversation still accumulates private cache. If the service reaches memory pressure, it must queue work, reject requests, evict reusable prefixes, move data to slower memory, shorten retained context, or reduce concurrency. Each choice changes latency or capability.

What KV caching does not solve

KV caching is often loosely described as model memory, but that framing conceals its limits.

• It does not create durable memory. Once the cache is discarded, the model does not retain the exchange unless information was stored elsewhere.
• It does not guarantee effective recall. A token being present in context does not ensure that the model will attend to it correctly.
• It does not remove sequential decoding. The next sampled token remains an input to the following step.
• It does not make context free. Longer sequences increase cache use and attention-reading work.
• It is model-specific state. KV tensors generally cannot be transferred casually between models, architectures, or incompatible configurations.
• It creates a security surface. Shared prefixes, eviction logic, and memory reuse require strict tenant isolation and careful clearing practices.

Nor is every cache entry equally valuable. Recent tokens may matter for local coherence, while selected earlier instructions or facts may remain crucial. Conventional caching preserves positions mechanically rather than according to future informational value.

The open frontier: deciding what deserves to remain

The deeper opportunity is not simply to store more tokens. It is to make retention selective without corrupting generation.

Researchers and system builders are exploring variations of sliding windows, token eviction, cache merging, recurrent summaries, offloading, and attention architectures whose cost does not grow in the same manner. Each raises difficult questions. Can a system identify dispensable tokens before future queries reveal their importance? Can compressed state preserve exact instructions, names, and code dependencies? When is moving cache to host memory beneficial, given transfer latency? How should a scheduler balance one very long request against many short ones?

There is also a product-level question: should a conversation remain verbatim in active context, or should older material be transformed into structured state? KV caching makes verbatim continuation efficient within limits. Databases, retrieval systems, and summaries serve different forms of continuity. Mature AI products will likely coordinate all of them rather than treating a maximum context window as a complete memory architecture.

The KV cache is therefore more than an inference trick. It is the hidden ledger of what a running model can still attend to cheaply. Its design determines how quickly an answer begins, how smoothly it unfolds, how many users can be served, and how expensive long context becomes. The next advances in AI interaction may depend less on expanding that ledger than on learning what can be safely forgotten.

This post was drafted with AI assistance and reviewed against our editorial policy before publication. Corrections are made at the source, on the page, with the date shown.