KV Cache is a key concept connecting “Transformer theory” with “LLM engineering and deployment.” Understanding it completes the final link between “how a model computes” and “how a model runs.”

KV Cache in Depth: From Transformer Fundamentals to API Cost Optimization

A conceptual guide, first-principles explanation, and practical engineering reference for AI application engineers

1. What Is KV Cache? Start with Why It Is Needed

1.1 The Central Tension in Transformer Inference

LLMs such as GPT and Claude use a decoder-only Transformer architecture. During inference, the model generates text autoregressively: it produces one token at a time, appends it to the end of the sequence, and then generates the next token. This repeats until an end token is produced.

At each generation step, Self-Attention requires the current token’s Query to take a dot product with the Keys of all existing tokens in the sequence, and then compute a weighted sum of all existing Values. This is the practical meaning of the Attention formula $ ext{softmax}( rac{QK^T}{sqrt{d_k}})V$.

The Transformer model itself, however, is stateless. Each forward pass behaves like a pure function: it receives an input, computes a result, returns the output, and then all intermediate results—including every token’s K and V vectors—disappear. The model does not automatically “remember” what it calculated in the previous forward pass.

Without optimization, this means that every time a new token is generated, the model must feed the entire sequence, including all previously processed tokens, through the model again and recompute the K and V for every token. As the sequence grows, the amount of repeated computation grows quadratically, which is unacceptable in engineering practice.

1.2 The KV Cache Solution

The central idea of KV Cache is straightforward: cache the Key and Value vectors already computed at each step in GPU memory, and reuse them in the next step to avoid redundant computation.

In essence, KV Cache adds state to a stateless model, allowing it to “remember” results calculated in previous steps.

1.3 A Concrete Example

Suppose the model needs to generate “The cat sat on the mat.”

Step 1: the prompt is “The cat sat on the” at positions 1–5. The model computes the K and V for all five tokens in parallel, completes the attention calculation, sends the output at position 5 to the prediction head, and predicts that the next token is “mat.”

Step 2 without KV Cache: when predicting the next token at position 7, the current sequence is “The cat sat on the mat,” containing six tokens. The model must feed all six tokens through again, recompute the K and V for every token, and perform the full attention calculation. But the K and V at positions 1–5 are exactly the same as in Step 1 because neither the model weights nor the inputs have changed. Recomputing them is pure waste.

Step 2 with KV Cache: the K and V at positions 1–5 are already cached in GPU memory. To predict the next token at position 7, the model only needs to feed in the single token “mat,” compute its Q, K, and V, append the new K and V to the cache, and use its Q to attend to all six cached Keys, including the K for “mat” itself (see Note 1). The amount of computation falls from processing six tokens to processing one.

Note 1:

The precise definition of the Causal Mask

The Causal Mask rule is: position $i$ can see positions 1, 2, …, $i-1$, and $i$. In other words, position $i$ can see itself.

Why must it see itself?

Consider the simplest case: position 1, the first token in the sequence.

If the causal mask allowed position $i$ to see only positions 1 through $i-1$, then position 1 would be able to see positions 1 through 0—in other words, nothing. The token’s attention output would be empty and contain no information, so the model could not perform any meaningful computation from it.

In reality, the Q at position 1 must take a dot product with its own K to obtain its attention weight to itself. When there is only one visible token, that weight is 1.0. It then uses its own V as the attention output, allowing the information at position 1 to propagate normally.

The complete information flow is therefore as follows.

In one Transformer layer, the attention output at position $i$ is:

$$ o_i = \sum_{j=1}^{i} \alpha_{ij} \cdot v_j $$

The summation runs from $j = 1$ through $j = i$, including $i$ itself. $alpha_{ii}$, the attention weight from position $i$ to itself, is normally nonzero. Every token’s output therefore includes a contribution from its own V.

Thus, when we say that “the hidden state at position $i$ combines information from positions 1 through $i$,” the $i$ is exact—not $i-1$.

2. The Exact Location of KV Cache in the Transformer Architecture

2.1 Where It Exists

KV Cache exists in every attention head of every Transformer layer.

During inference, a typical LLM—for example, one with 32 layers, 32 attention heads, and a head dimension of 128—must store a K matrix and a V matrix for every layer and every head. For a context with sequence length $n$, the total KV Cache size is:

$$ 2 \times n_{\text{layers}} \times n_{\text{heads}} \times n_{\text{seq}} \times d_{\text{head}} \times \text{bytes\_per\_element} $$

This is why long context windows of 128K or 1M tokens pose such a major challenge for GPU memory. KV Cache usage grows linearly with sequence length and consumes memory in addition to the model parameters.

2.2 The Two Phases of Inference

Understanding KV Cache requires distinguishing two fundamentally different phases of inference:

Prefill phase: the user’s prompt enters the model, which processes all prompt tokens in parallel and computes and stores their K and V for every layer and head. This phase is compute-bound, similar to a forward pass during training. Its duration determines Time to First Token (TTFT)—the time a user waits after submitting a request before the first token appears.

Decode phase: output tokens are generated one at a time. Each step computes the Q, K, and V for only one token, appends the new K and V to the cache, and uses the new Q to query the complete cache. This phase is memory-bandwidth-bound, because every step must read the entire KV Cache from GPU memory.

2.3 What Actually Happens During Each Prediction Step

One easily confused point needs clarification.

When we say “predict token N+1,” the model actually computes the output representation at position N, the final token in the sequence. The process is:

The token at position N passes through the Embedding layer to obtain vector $x_N$.
$x_N$ passes through every Transformer layer in sequence. In each layer it is projected by $W_Q$, $W_K$, and $W_V$ to produce $q_N$, $k_N$, and $v_N$.
$q_N$ takes dot products with all Keys at positions 1 through N to obtain attention weights, which are then used for a weighted sum of all Values.
After the attention output passes through components such as the FFN, it becomes the final hidden state $h_N$ at position N.
$h_N$ is sent to the model’s Output Head, a Linear layer followed by Softmax. The Linear layer has a weight matrix of shape $d_{\text{model}} \times V$, where $V$ is the vocabulary size. It maps $h_N$ to a $V$-dimensional logit vector, which Softmax converts into a probability distribution. Sampling or argmax then selects token N+1.

The key insight is that “predicting token N+1” and “computing the output at position N” are the same operation. Token N+1 does not exist before it is predicted, so there can be no “Q at position N+1.” The Q always comes from the token at the final position of the current sequence.

To go further, the Causal Attention Mask ensures that position $i$ can see only information from positions 1 through $i$. Therefore, $h_i$ encodes the semantics of “the most likely next token after all tokens from positions 1 through $i$.” During training, the model processes the entire sequence at once, every position produces a prediction simultaneously, and a loss can be computed at every position. This is why one training sample provides multiple training signals. During inference, we care only about the output at the last position, because only that position predicts the genuinely unseen next token.

2.4 Why Cache K and V, but Not Q?

At each decode step, we only need the Q at the current final position to query the K and V at every historical position. That Q is computed in real time during the step from the new token just appended to the sequence. What about the old Q vectors at previous positions? Each already fulfilled its purpose in its own step—predicting the token at the following position—and is not needed at all in the current step. By contrast, every position’s K and V must be accessed at every step because the current Q takes dot products with all historical Keys and computes a weighted sum of all historical Values. They must therefore be cached.

3. From KV Cache Within One Request to Prompt Caching Across Requests

3.1 Two Distinct Layers

This distinction is essential:

Layer 1: KV Cache within one request—an optimization inside the Transformer inference engine. During the decode phase of one API request, it avoids recomputing the K and V of every old token whenever a new token is generated. It is completely transparent and invisible to API users, and is handled automatically by inference frameworks such as vLLM and TensorRT-LLM.

Layer 2: Prompt Caching across requests—an infrastructure-level optimization implemented by API providers. When one API request ends, the server does not discard the KV Cache computed during that request’s prefill phase, but retains it for a period of time. If a later request has the same prompt prefix, the server can load the existing KV Cache directly and skip prefill computation for the repeated prefix.

3.2 How Cross-Request Prompt Caching Works

Consider a multi-turn question-answering scenario over a long document.

The first request has this prompt structure: [system instructions] + [50,000-token document] + [user question A]

The second request has this prompt structure: [system instructions] + [50,000-token document] + [user question B]

The first 50,000-plus tokens are identical in both requests; only the final user question differs.

Without Prompt Caching: during the second request’s prefill phase, the model recomputes all 50,000-plus tokens from scratch. You pay full price twice for exactly the same computation.

With Prompt Caching: the server retains the KV Cache from the first request. When the second request arrives, the system recognizes the identical prefix, loads the existing cache, performs prefill only for the few dozen tokens in the new user question, and then enters the decode phase.

3.3 Why It Must Be an “Identical Prefix” (Note 2)

This requirement follows from the nature of the Causal Mask. In causal attention, the K and V at position $i$ depend only on the inputs at positions 1 through $i$. Therefore, if the first $n$ prompt tokens are completely identical, the K and V calculated for those $n$ tokens at every layer are guaranteed to be identical, regardless of what content follows them.

But if even one token in the prefix differs—for example, because a timestamp was inserted at the beginning—every subsequent K and V changes from that position onward, invalidating the rest of the cache.

Note 2:

What is an “identical prefix”?

A prefix is the continuous identical part of the prompt starting from the beginning. As long as two requests match token by token from the first token onward, the continuous matching part is the “identical prefix.” The cache becomes invalid at the first different token.

Example 1: a typical cache-hit scenario

First request prompt:

[System] You are a legal-document analysis assistant. Answer questions based on the following contract.
[Full contract] (50,000 tokens)
[User] What are the termination clauses in this contract?

Second request prompt:

[System] You are a legal-document analysis assistant. Answer questions based on the following contract.
[Full contract] (50,000 tokens, exactly the same as before)
[User] What is the payment cycle specified in the contract?

Compare token by token from the beginning: the System instructions match, and the full contract matches, until the user questions diverge. The first roughly 50,000-plus tokens form an identical prefix, producing a cache hit. The model only needs to prefill the final, different user question.

Example 2: adding a timestamp at the beginning completely invalidates the cache

First request:

[timestamp: 2025-03-05 10:00:00]
[System] You are a legal-document analysis assistant...
[Full contract] (50,000 tokens)
[User] What are the termination clauses?

Second request:

[timestamp: 2025-03-05 10:02:35]
[System] You are a legal-document analysis assistant...
[Full contract] (50,000 tokens, exactly the same)
[User] What is the payment cycle?

Starting from the first token, 2025-03-05 10:00:00 differs from 2025-03-05 10:02:35 at the timestamp’s seconds. The identical prefix is almost zero tokens long. Even though 50,000 subsequent tokens are identical, they do not help because prefix matching broke at the very beginning. The entire prompt must be recomputed at full cost.

This is why cost-optimization guidance repeatedly emphasizes: never place dynamically changing content at the beginning of a prompt.

Example 3: changing one word in the middle hits the first half and invalidates the second

First request:

[System] You are a professional document-analysis assistant.
[Document A] (20,000 tokens)
[Document B] (30,000 tokens)
[User] Summarize the key points of Document B.

Second request:

[System] You are a professional document-analysis assistant.
[Document A] (20,000 tokens, exactly the same)
[Document C] (30,000 tokens, different from Document B)
[User] Summarize the key points of Document C.

Comparing token by token from the beginning, the System instructions and Document A match. The divergence begins at Document B versus Document C. The identical prefix is roughly 20,000-plus tokens—the System and Document A—and that portion hits the cache. Everything from Document B or C onward, more than 30,000 tokens, must be prefilled again.

Example 4: changing the order of few-shot examples invalidates almost everything

First request:

[System] You are a sentiment-analysis assistant.
[Example 1] Input: "This product is excellent" → Output: Positive
[Example 2] Input: "The service is terrible" → Output: Negative
[Example 3] Input: "It's okay" → Output: Neutral
[User] Analyze: "I'm in a good mood today"

Second request:

[System] You are a sentiment-analysis assistant.
[Example 2] Input: "The service is terrible" → Output: Negative
[Example 1] Input: "This product is excellent" → Output: Positive
[Example 3] Input: "It's okay" → Output: Neutral
[User] Analyze: "The price is too high"

The System instructions match, but Example 1 and Example 2 differ immediately afterward. The identical prefix contains only the short System section. Even though the examples have exactly the same content and only their order changed, almost the entire cache becomes unusable.

Engineering implication: once the order of few-shot examples has been established, do not randomize it, or you will destroy the cache.

In one sentence:

Prefix matching is like aligning two ropes from their beginnings: the portion before the first fork is a cache-hit region, while all content after that fork must be recomputed, no matter how similar it is. The prompt-design principle is therefore to place as much invariant content as possible near the beginning and as much variable content as possible near the end.

4. Prompt Caching Pricing Across Three Major API Providers (as of 2025)

4.1 Anthropic (Claude): Explicit Control and High Hit Rates

Mechanism: requests must explicitly mark cache breakpoints with the cache_control parameter.
Pricing: writing to a five-minute cache costs 1.25x the base input-token price, while cache reads cost only 0.1x, or 10%, of the base price. A one-hour cache option is also available, with higher write cost but longer retention.
Minimum cache size: Claude Sonnet 4.5, Opus 4, Sonnet 4, and similar models require 1,024 tokens.
Invalidation logic: request components are processed in the fixed order Tools → System Message → Message History. A change to an earlier component invalidates the following cache.
Observed hit rate: when caching is marked proactively, the hit rate approaches 100%, making performance predictable.
Break-even point: because a cache write has a 1.25x premium, at least two requests are needed to break even.

4.2 OpenAI (GPT): Fully Automatic, Zero Configuration

Mechanism: applies automatically to gpt-4o and newer models, requiring no code changes and no additional fees.
Pricing: cache hits receive discounts of up to 90% on the latest models, with no additional cache-write charge.
Minimum cache size: 1,024 tokens, matched in increments of 128 tokens.
Cache lifetime: typically cleared after 5–10 minutes of inactivity, with a maximum of one hour. The prompt_cache_key parameter can assist routing.
Observed hit rate: because routing is automatic, the hit rate is approximately 50%, so performance may be less consistent.

4.3 Google (Gemini): Two Mechanisms with Time-Based Charges

Mechanism: two modes are available. Implicit caching is enabled by default, automatically detects repeated prefixes, and applies a discount. Explicit caching requires manually creating a cache through the API and controlling its TTL.
Pricing: for Gemini 2.5 and later models, reading cached tokens costs 10% of the standard input price, a 90% discount. Explicit caching adds time-based storage charges of approximately $4.50 per million tokens per hour.
TTL: explicit caching defaults to 60 minutes and is configurable. Implicit caching has no storage cost.
Use case: explicit caching is better suited to repeated queries against the same large document over a longer period.

4.4 Summary of the Main Differences

Dimension	Anthropic	OpenAI	Google Gemini
Control	Explicit marker	Fully automatic	Implicit automatic + explicit manual
Cache-write premium	Yes (1.25x)	None	None, but explicit caching has storage fees
Cache-read discount	90% off	Up to 90% off	90% off
Hit-rate control	High (close to 100%)	Low (about 50%)	Medium
Minimum token count	1,024	1,024	Model-dependent (minimum 1,024)
Storage fees	None	None	Time-based fees for explicit caching

5. A Cost-Optimization Framework for AI Application Engineers

5.1 Layer 1: Think in Terms of Prompt Structure

Treat a prompt as a layered structure consisting of a “stable prefix + dynamic suffix”:

Stable prefix, placed first: system instructions, tool definitions, few-shot examples, and long-document context—content that remains unchanged across requests.
Dynamic suffix, placed last: the user’s specific question, the current conversation turn, and other content that differs on every request.

A common anti-pattern is placing timestamps, request IDs, or other changing metadata at the beginning of a prompt. This makes the prefix differ from the first token and completely invalidates the cache.

5.2 Layer 2: Match the Request Pattern

Choose the most suitable provider and caching strategy for the application:

High-frequency scenarios where many users share the same system prompt, such as a customer-service bot, fit OpenAI’s automatic caching and benefit with zero configuration.
Multi-turn questions from one user about the same long document fit Anthropic’s or Gemini’s explicit caching, which ensures a high hit rate.
Asynchronous batch processing can combine Batch API discounts, usually 50%, with Prompt Caching discounts for two layers of savings.

5.3 Layer 3: Think in Full-Pipeline Token Economics

Do not focus only on the cost of a single request. Calculate the complete task pipeline:

How many input and output tokens does one user’s full task consume?
How much of that content repeats across requests?
What cache hit rate can you achieve?
Where is the break-even point? Anthropic cache writes carry a 1.25x premium, so at least two requests are needed to recover the cost.

5.4 Layer 4: Think at the Architectural Cost Level

Model Routing: use inexpensive models for simple tasks and costly models only for complex tasks.
Context Compaction: summarize long conversation history to prevent unbounded context-window growth.
Monitoring and validation: monitor actual cache metrics to confirm that the optimization works. Anthropic returns cache_read_input_tokens and cache_creation_input_tokens; OpenAI returns cached_tokens; Gemini returns cachedContentTokenCount.

5.5 Layer 5: Develop Engineering Intuition for the Underlying Constraints

Understand the fundamental reason providers can offer a 90% discount. On a cache hit, they skip much of the matrix multiplication in the prefill phase and only need to read existing KV vectors from GPU memory. Those vectors still occupy GPU memory, so providers must balance memory cost against discount size. This explains why caches have TTL limits and minimum token-count requirements, and why Google’s explicit caching charges according to storage duration.

6. Panoramic Knowledge Map

Transformer Self-Attention (computing Q, K, and V)
    │
    ├─→ Training: all tokens computed in parallel; no KV Cache concept
    │
    └─→ Inference: autoregressive generation, output one token at a time
         │
         ├─→ Prefill phase (process prompt; compute-bound; determines TTFT)
         │    └─ Compute K/V for all prompt tokens at once and store them in cache
         │
         └─→ Decode phase (generate token by token; memory-bandwidth-bound)
              └─ At each step, compute only the new token's Q/K/V; Q queries all cached K/V
                   │
                   └─→ [Layer 1] KV Cache within one request
                        (handled automatically by inference engine; invisible to API users)
                        │
                        └─→ Request ends → KV Cache is normally discarded
                             │
                             └─→ [Layer 2] Prompt Caching across requests
                                  (API provider retains KV Cache for later reuse)
                                  │
                                  ├─ Anthropic: explicit control, cache write 1.25x, read 0.1x
                                  ├─ OpenAI: fully automatic, up to 90% off, uncontrollable hit rate
                                  └─ Google: implicit + explicit, 90% off, explicit storage fee
                                       │
                                       └─→ Cost optimization for AI application engineers
                                            ├─ Prompt structure: stable prefix + dynamic suffix
                                            ├─ Request pattern: choose a strategy for the scenario
                                            ├─ Token economics: calculate full-pipeline cost
                                            ├─ Architecture: routing + compaction + monitoring
                                            └─ Underlying intuition: understand resource constraints behind discounts

KV Cache in Depth: From Transformer Fundamentals to API Cost Optimization#

1. What Is KV Cache? Start with Why It Is Needed#

1.1 The Central Tension in Transformer Inference#

1.2 The KV Cache Solution#

1.3 A Concrete Example#

2. The Exact Location of KV Cache in the Transformer Architecture#

2.1 Where It Exists#

2.2 The Two Phases of Inference#

2.3 What Actually Happens During Each Prediction Step#

2.4 Why Cache K and V, but Not Q?#

3. From KV Cache Within One Request to Prompt Caching Across Requests#

3.1 Two Distinct Layers#

3.2 How Cross-Request Prompt Caching Works#

3.3 Why It Must Be an “Identical Prefix” (Note 2)#

4. Prompt Caching Pricing Across Three Major API Providers (as of 2025)#

4.1 Anthropic (Claude): Explicit Control and High Hit Rates#

4.2 OpenAI (GPT): Fully Automatic, Zero Configuration#

4.3 Google (Gemini): Two Mechanisms with Time-Based Charges#

4.4 Summary of the Main Differences#

5. A Cost-Optimization Framework for AI Application Engineers#

5.1 Layer 1: Think in Terms of Prompt Structure#

5.2 Layer 2: Match the Request Pattern#

5.3 Layer 3: Think in Full-Pipeline Token Economics#

5.4 Layer 4: Think at the Architectural Cost Level#

5.5 Layer 5: Develop Engineering Intuition for the Underlying Constraints#

6. Panoramic Knowledge Map#

KV Cache in Depth: From Transformer Fundamentals to API Cost Optimization

1. What Is KV Cache? Start with Why It Is Needed

1.1 The Central Tension in Transformer Inference

1.2 The KV Cache Solution

1.3 A Concrete Example

2. The Exact Location of KV Cache in the Transformer Architecture

2.1 Where It Exists

2.2 The Two Phases of Inference

2.3 What Actually Happens During Each Prediction Step

2.4 Why Cache K and V, but Not Q?

3. From KV Cache Within One Request to Prompt Caching Across Requests

3.1 Two Distinct Layers

3.2 How Cross-Request Prompt Caching Works

3.3 Why It Must Be an “Identical Prefix” (Note 2)

4. Prompt Caching Pricing Across Three Major API Providers (as of 2025)

4.1 Anthropic (Claude): Explicit Control and High Hit Rates

4.2 OpenAI (GPT): Fully Automatic, Zero Configuration

4.3 Google (Gemini): Two Mechanisms with Time-Based Charges

4.4 Summary of the Main Differences

5. A Cost-Optimization Framework for AI Application Engineers

5.1 Layer 1: Think in Terms of Prompt Structure

5.2 Layer 2: Match the Request Pattern

5.3 Layer 3: Think in Full-Pipeline Token Economics

5.4 Layer 4: Think at the Architectural Cost Level

5.5 Layer 5: Develop Engineering Intuition for the Underlying Constraints

6. Panoramic Knowledge Map