RAG Chunking Best-Practices Guide
Compiled from benchmark data and industry practices published by sources including NVIDIA, Microsoft Azure, Vectara (NAACL 2025), Chroma Research, and FloTorch (2026.02).
Last updated: March 2026
1. Why Chunking Is the Most Critical Stage in RAG
Eighty percent of RAG failures can be traced to the ingestion and chunking layer rather than the LLM itself.
Research that Vectara presented at NAACL 2025 (arXiv:2410.13070) cross-tested 25 chunking configurations with 48 embedding models. It concluded that the chunking configuration affects retrieval quality as much as, or even more than, the choice of embedding model. In Chroma’s evaluation, the recall gap between the best and worst chunking strategies on the same corpus reached 9%.
The central tension is this: if a chunk is too large, its embedding becomes diluted and retrieval loses precision; if it is too small, context is lost and the retrieved result is no longer useful. Every strategy seeks a balance within this trade-off.
2. Recommended Defaults: Start Here
| Parameter | Recommended Starting Value | Source |
|---|---|---|
| Strategy | Recursive Character Splitting | Ranked first in the FloTorch 2026 benchmark |
| Chunk Size | 512 tokens | Consensus among Microsoft Azure, NVIDIA, and FloTorch |
| Overlap | 50–100 tokens (10–20%) | NVIDIA found 15% optimal; Azure recommends 25% (128 tokens) |
| Maximum chunk limit | 1024 tokens | Split again when this length is exceeded |
| Oversized paragraphs | Recursive fallback splitting | Built into recursive splitting |
These defaults have been validated by benchmarks. In FloTorch’s February 2026 test, recursive 512-token splitting ranked first among seven strategies with 69% end-to-end accuracy and required no additional model calls.
Consider a more complex strategy only when evaluation metrics for this baseline reach a ceiling.
3. Comparing Six Mainstream Chunking Strategies
3.1 Fixed-Size Chunking
Split at a fixed token or character count without considering semantics.
- Advantages: Simplest implementation, predictable behavior, and fastest indexing
- Disadvantages: Can split a sentence or paragraph in the middle and destroy semantic integrity
- Suitable for: Log files, uniformly structured data, and rapid prototypes
3.2 Recursive Character Splitting ⭐ Recommended Default
Recursively split with a sequence of separators in descending priority: ["\n\n", "\n", " ", ""]
First try paragraph boundaries → if a block is still too large, split by line → if still too large, split by spaces → as a final fallback, split by character. Each level processes only the oversized blocks left by the previous one.
- Advantages: Respects document structure, controls chunk size, requires no additional model calls, and is the most stable option in benchmarks
- Disadvantages: Does not understand semantics; it considers only formatting boundaries
- Suitable for: The preferred starting point for most general RAG workloads
Variant for code files: prioritize class definitions → function definitions → paragraphs → lines.
3.3 Sentence-Based Chunking
Use an NLP sentence segmenter such as spaCy or NLTK to split a document into sentences, then combine sentences up to the target length.
- Advantages: Never splits a sentence in half
- Disadvantages: Can still break cross-sentence meaning
- Suitable for: FAQ documents and documents with short paragraphs
A systematic analysis from January 2026 showed that sentence chunking performs on par with semantic chunking for content under 5,000 tokens while costing only a fraction as much.
3.4 Semantic Chunking
Use embeddings to calculate the semantic similarity of adjacent sentences and split where similarity drops sharply.
- Advantages: Chunk boundaries genuinely align with topic boundaries
- Disadvantages:
- High computational cost, because every pair of adjacent sentences requires an embedding calculation
- Unpredictable chunk sizes
- Only 54% accuracy in the FloTorch 2026 test because it produced fragments averaging just 43 tokens
- Suitable for: Scenarios with extremely high accuracy requirements, sufficient budget, and documents that change topics frequently
- Note: Chroma’s research shows that semantic chunking’s recall is only 2–3% higher than recursive splitting (91–92% versus 85–90%), despite costing much more
3.5 Document Structure-Based Chunking
Use structural signals already present in a document, such as Markdown headings, HTML tags, PDF sections, or a code AST.
- Advantages: Performs best on structured documents because the author has already grouped the semantics
- Disadvantages: Depends on good document structure and is unsuitable for unstructured text
- Suitable for: Technical documentation, API documentation, legal contracts, and academic papers
NVIDIA’s tests show that page-level chunking is the most stable option for queries requiring complex analytical reasoning, such as those over financial documents.
3.6 Contextual Chunking
Anthropic’s method uses an LLM to generate a contextual prefix for each chunk automatically, describing the chunk’s position and background within the whole document.
- Advantages: Significantly improves retrieval accuracy; Anthropic reports a 49% reduction in retrieval failures
- Disadvantages: Every chunk requires an LLM call, making batch processing expensive
- Suitable for: Production systems with extremely high accuracy requirements and sufficient budget
It can be combined with any splitting strategy—for example, first split recursively, then use an LLM to add a contextual prefix.
4. Detailed Tuning Guidance for Three Core Parameters
4.1 Chunk Size
Rule of thumb: a chunk should be able to answer one question independently or provide the complete information needed to answer it.
| Query Type | Recommended Chunk Size | Source |
|---|---|---|
| Factual query (“Where do I get the API key for XX?”) | 256–512 tokens | NVIDIA: DigitalCorpora / Earnings datasets |
| Analytical query (“Compare the Q3 and Q4 revenue trends”) | 512–1024 tokens | NVIDIA: FinanceBench dataset |
| General mixed workload | 512 tokens | Consensus starting point across multiple benchmarks |
Be aware of the “context cliff.” A systematic analysis in January 2026 found that LLM response quality declines noticeably once an individual chunk exceeds roughly 2,500 tokens. Even if a model supports a 128K context window, do not use enormous chunks.
Align with the embedding model: BGE-M3 accepts up to 8,192 tokens, but 512–1,024 tokens performs better in practice. The embedding model’s maximum input length is a hard ceiling, not a target.
4.2 Overlap
Rule of thumb: 10–20% of the chunk size.
| Chunk Size | Recommended Overlap | Notes |
|---|---|---|
| 256 tokens | 25–50 tokens | |
| 512 tokens | 50–100 tokens | |
| 1024 tokens | 100–150 tokens |
NVIDIA tested values of 10%, 15%, and 20% on FinanceBench; 15% performed best.
One disputed finding deserves attention: a January 2026 systematic analysis using SPLADE retrieval with Mistral-8B found that overlap produced no measurable benefit and only increased indexing cost. This reminds us that overlap depends on the retrieval method and characteristics of the data. Do not assume blindly that overlap is always useful; verify it through evaluation.
Side effects of excessive overlap include storage growth, duplicate content wasting context-window space, and a larger embedding index slowing queries.
4.3 Handling Oversized Paragraphs
When a single natural paragraph or code block exceeds the target chunk size:
- Split it internally with the primary strategy’s recursive fallback mechanism.
- Preserve the primary strategy’s overlap setting during the split.
- Preserve the parent metadata on each child chunk, including the original paragraph’s heading and position, to support subsequent reconstruction.
5. Choosing a Strategy for Different Document Types
| Document Type | Recommended Strategy | Explanation |
|---|---|---|
| Markdown/HTML technical documentation | Structure-Based + Recursive fallback | Split by heading level, then recursively split oversized sections |
| Plain text | Recursive Character Splitting | General-purpose default |
| Structured PDF | Structure-Based (sections/headings) | Extract structure with Document Intelligence first |
| Scanned/unstructured PDF | OCR → Sentence-Based or Recursive | Run OCR before splitting |
| Code files | AST-Based / Code-Aware Recursive | Split at class → function → method boundaries |
| Tabular data | Row-level/cell-level splitting | Use each row or logical group of rows as a chunk |
| FAQ / Q&A documents | One Q&A pair per chunk | Natural boundaries are explicit |
| Legal/financial documents | Page-Level or Section-Based | Page-level was most stable for these documents in NVIDIA’s tests |
A production system should have a chunking router that selects a strategy automatically based on file type and document structure instead of applying one strategy to every document.
6. Metadata That Must Be Preserved
In addition to the text itself, each chunk should store:
| Metadata Field | Purpose |
|---|---|
source_file | Source file path/name |
page_number / line_range | Exact location in the original document |
heading_hierarchy | Heading hierarchy, such as “Chapter 3 > 3.2 Authentication > JWT” |
chunk_index | The chunk’s sequential position within the document |
total_chunks | Total number of chunks in the document |
doc_type | Document type, used for retrieval filters |
created_at / updated_at | Timestamps, used to rank by freshness |
language | Language identifier, required for multilingual workloads |
Why metadata matters: filter retrieval by source or doc_type, provide citations when presenting results, and reconstruct context from neighboring chunks.
7. An Evaluation-Driven Tuning Workflow
Do not choose parameters by intuition. Use the following workflow.
Step 1: Build an Evaluation Set
Prepare 50–100 (question, expected_answer, source_document) tuples covering typical query patterns.
Step 2: Establish a Baseline
Run the recommended defaults—recursive, 512 tokens, and overlap of 50–100—and record the baseline metrics.
Step 3: Core Evaluation Metrics
| Metric | Meaning | Tool |
|---|---|---|
| Context Recall | How much of the required information appears in the retrieved chunks | Ragas / DeepEval |
| Context Precision | How many retrieved chunks are genuinely relevant rather than noise | Ragas / DeepEval |
| Answer Correctness | Correctness of the final generated answer | Ragas / DeepEval |
| Faithfulness | Whether the answer remains faithful to the retrieved context instead of hallucinating | Ragas / DeepEval |
Step 4: Parameter Sweep
Change only one parameter at a time:
- Chunk size: [256, 512, 768, 1024]
- Overlap: [0, 50, 100, 150]
- Strategy: [recursive, sentence, structure-based]
Step 5: Iterate
If context recall is low → check whether chunks are too small or the strategy splits essential information.
If context precision is low → check whether chunks are too large and include irrelevant content.
If faithfulness is low → the wrong chunks may have been retrieved; inspect embeddings and reranking.
8. Quality-Check Checklist
After chunking, run these checks:
- Are there many fragment chunks under 20 tokens? This usually indicates a bug in the splitting logic.
- Does any chunk exceed the embedding model’s maximum input length?
- Sample 20 chunks and inspect them manually for clear semantic breaks.
- Is the distribution of chunk token lengths reasonable? The histogram should be approximately normal without an extreme long tail.
- Is the metadata complete, with values for source, position, and heading?
- Does text in the overlap region align correctly with adjacent chunks?
9. Advanced Optimization Directions
Once baseline evaluation metrics reach a ceiling, consider the following options in priority order:
- Contextual Chunking (Anthropic’s method): use an LLM to add a contextual prefix to every chunk. It is expensive but produces a substantial improvement.
- Hybrid Retrieval: combine dense embedding retrieval with sparse BM25 retrieval; the combination works better than either method alone.
- Reranking: retrieve the top 20, use a reranker such as Cohere Rerank to reorder them, and send the top five to the LLM. Small, precise chunks provide a clearer signal to the reranker.
- Query Transformation: rewrite, expand, or decompose the user’s query before retrieval to improve hit rates.
- Late Chunking: embed the entire document before splitting it, preserving global semantic information. This is a newer method and remains experimental.
10. Common Pitfalls
Pitfall 1: Starting with semantic chunking The FloTorch 2026 test showed that semantic chunking achieved only 54% accuracy because it produced fragments that were too short, far below recursive splitting’s 69%. Start with a simple strategy and upgrade only when data demonstrates the need.
Pitfall 2: Using the embedding model’s maximum input length as the chunk size The fact that BGE-M3 accepts 8,192 tokens does not mean you should create 8,192-token chunks. In practice, 512–1,024-token chunks retrieve far better than chunks that fill the entire window.
Pitfall 3: Applying one strategy to every document Code, Markdown, PDF, and tables require completely different optimal splitting methods. A production system needs a chunking router.
Pitfall 4: Tuning parameters without evaluation Parameters selected by intuition are rarely optimal. Prepare an evaluation set and compare them quantitatively with Ragas or DeepEval.
Pitfall 5: Failing to preserve metadata after chunking Without metadata, you cannot filter by source, locate positions, or reconstruct neighboring chunks. Adding it later is extremely expensive.
Pitfall 6: Ignoring the storage cost of overlap A 20% overlap means the vector database must store roughly 20% more data. At corpus scale, this is not a small amount.
References
- NVIDIA Technical Blog: Finding the Best Chunking Strategy for Accurate AI Responses (2025)
- Vectara / FloTorch Benchmark: 50 academic papers, 7 strategies, February 2026
- Chroma Research: Token-level retrieval recall across chunking strategies (2025)
- Microsoft Azure Architecture Center: RAG Chunking Phase
- Anthropic: Contextual Retrieval Paper (2024)
- Databricks Technical Blog: The Ultimate Guide to Chunking Strategies for RAG (2025)
- Stack Overflow Blog: Breaking up is hard to do — Chunking in RAG applications (2024)
- Firecrawl: Best Chunking Strategies for RAG in 2026
- PremAI: RAG Chunking Strategies — The 2026 Benchmark Guide