Notes on Anthropic's Contextual Retrieval

Summary of Anthropic’s Contextual Retrieval

1. What Is the Article Actually Saying?

Anthropic identified a fundamental weakness in traditional RAG: a chunk loses meaning when separated from its source document.

The classic example is a chunk that says, "The company's revenue grew by 3% over the previous quarter." Which company? Which quarter? That information was removed by chunking, so vector-similarity retrieval can no longer use it.

The proposed solution is remarkably simple: before embedding, use an LLM to add a short contextual prefix to every chunk. Feed the full document and the current chunk to Claude, ask it to generate a 50–100-token description of the chunk’s context, prepend that description to the chunk, and then create the embedding.

2. What Does the Data in Appendix II Tell Us?

The PDF presents cross-domain evaluation examples covering codebases, novels, academic papers, and ArXiv papers. Several observations stand out.

1. Evaluation-dataset design is the essential part

Every example contains a Query → Golden Answer → Golden Chunk → Context tuple. This is deliberate. A RAG evaluation needs ground truth identifying which chunk should contain the correct answer. Without that relationship, the metrics are calculated against nothing meaningful.

2. The selected domains are representative

Codebases require precise matching, novels require cross-paragraph understanding of characters and relationships, and academic papers contain specialized terminology and citation chains. Each domain exposes a different weakness in RAG. A serious project should evaluate at least two different kinds of documents rather than relying on only one.

3. The Context field is the core output of Contextual Retrieval

Look at the Context field in each PDF example. It is a natural-language explanation of the chunk’s place and role in the full document. In the codebase example, the context explains that the code implements the BasicConfigurator class in the Log4cxx logging library. Once this information is prepended to the chunk, its embedding can capture the meaning “code related to logging configuration” instead of representing only a collection of function signatures.

3. Quantitative Results

These numbers provide a useful benchmark for evaluating a RAG implementation.

4. My View of RAG Best Practices

Based on Contextual Retrieval and production experience through 2026, this is the complete RAG architecture I would recommend.

Principle One: First Decide Whether You Really Need RAG

Anthropic is explicit: if the knowledge base contains fewer than 200K tokens, roughly 500 pages, putting the entire text into a prompt with prompt caching may be simpler and more accurate than RAG. Context windows are already large in 2026, so many use cases do not require RAG at all.

A good answer to “When should I use RAG?” is: use long context with prompt caching for a small knowledge base; use RAG for a large knowledge base or frequently updated data; use fine-tuning when the model’s behavioral pattern must change. Production systems often combine these approaches.

Principle Two: Retrieval Quality Determines Everything

Anthropic’s data demonstrates that every improvement at the retrieval layer produces a measurable return. A system should spend most of its optimization effort on retrieval rather than prompt tuning. Even excellent generation cannot rescue irrelevant evidence.

Recommended retrieval pipeline, added in layers:

Layer 1: Contextual Chunking (required)
  Use an LLM to add a contextual prefix to every chunk
  → Reduces retrieval failures by 35% on its own

Layer 2: Hybrid Search (required)
  Vector retrieval (semantic matching) + BM25 (exact keywords) + RRF fusion
  → Addresses the tendency of vector retrieval to capture meaning but miss keywords

Layer 3: Reranking (strongly recommended)
  Rerank top-k results with a cross-encoder or Cohere Reranker
  → Reduces total retrieval failures from 2.9% to 1.9%

Layer 4: Query Transformation (scenario-dependent)
  Query rewriting / HyDE / Multi-query
  → Handles vague queries and unclear user wording

Principle Three: Control the Cost of Contextual Retrieval

Generating context with an LLM for every chunk sounds expensive. Anthropic’s solution is prompt caching: cache the complete document once, then pay only for incremental tokens when generating the context for each chunk. With an 800-token chunk and an 8K-token document, their estimate is $1.02 per million document tokens. This is practical in production because it is a one-time indexing cost, not a runtime cost.

Implementation outline:

# Pseudocode: the core of Contextual Retrieval
CONTEXT_PROMPT = """
<document>
{whole_document}
</document>
<chunk>
{chunk_content}
</chunk>
Briefly describe the position and role of this chunk within the full document.
Output only the contextual description and nothing else.
"""

# 1. Cache the complete document once with prompt caching.
# 2. Generate context for each chunk in the document, hitting the cache.
# 3. contextualized_chunk = context + "\n\n" + original_chunk
# 4. Build embedding and BM25 indexes from contextualized_chunk.

Principle Four: Evaluation Requires Ground Truth

The PDF’s evaluation method implies that an evaluation dataset must contain:

• Query — a question a user might ask
• Golden Chunk — the chunk that should contain the answer
• Golden Answer — the expected answer

Without this, metrics from tools such as Ragas and DeepEval have no solid foundation. A useful evaluation set should contain at least 50–100 such examples.

Principle Five: Chunking Is Not One-Size-Fits-All

The PDF evaluates code, novels, and papers because each type needs a different chunking strategy.

• Code — chunk by function or class to preserve logical integrity
• Structured documents — chunk by section or paragraph to respect document structure
• Unstructured long text — use semantic chunking based on similarity thresholds

Principle Six: Choose Top-K Deliberately

Anthropic tested top-5, top-10, and top-20 settings and found that top-20 combined with reranking worked best. The idea is to retrieve a broad candidate set first, then let a reranker choose the strongest evidence.

My recommendation: retrieve the top 20 chunks, rerank them, and send the best five to the LLM. This preserves recall while controlling context length and cost.