Best Practices for Chunking in RAG Systems: From Fixed Chunks to Structural, Contextualized, and Evaluation-Driven Design

Chunking is a central design choice in the data-ingestion stage of RAG. It determines:

• what the retrieval system can find;
• whether an embedding represents one complete concept or a mixture of topics;
• whether citations can point to an exact location;
• how much context rerankers and LLMs must process;
• the system’s eventual accuracy, latency, and cost.

There is currently no universally optimal chunk size for every RAG system, nor one chunking algorithm that wins on every dataset. A more robust engineering conclusion is:

Use document structure and semantic completeness as the primary splitting criteria, treat token length as a constraint, and determine the final design with real retrieval and question-answering datasets.

Google Cloud explicitly states that there is no standard answer for the best chunking strategy because it depends on the data and application. A systematic 2026 study also found that chunking performance depends significantly on the retrieval task, dataset, and embedding method, and that complex LLM-driven methods do not necessarily outperform simple structural splitting. (Google Cloud Documentation)

1. What Problem Does Chunking Actually Solve?

An ideal chunk should satisfy three goals at once:

Semantically complete:
Contains a fact, concept, or rule that can be understood independently

Focused for retrieval:
Does not mix in large amounts of material unrelated to the query

Sufficient context:
Preserves the headings, qualifications, and relationships needed to understand the fact

These goals, however, are in tension with one another.

Chunks That Are Too Small

They may cause:

• subjects and conclusions to be separated;
• rules and exceptions to be separated;
• table data to lose its headers;
• pronouns to lose their referents;
• retrieval to find a local sentence that is insufficient to answer the question.

Chunks That Are Too Large

They may cause:

• one vector to represent several topics simultaneously;
• relevant information to be diluted by large amounts of irrelevant content;
• reranking and generation costs to rise;
• citation ranges to become too broad;
• embeddings to lose local detail.

Google’s current documentation summarizes the trade-off this way: smaller chunks usually produce more precise vector representations, while larger chunks preserve more context but are also more likely to miss local details or produce overly broad representations. (Google Cloud Documentation)

The goal of chunking is therefore not to make every block exactly the same length. It is to:

Divide semantic units into sizes suitable for use as retrieval objects.

2. A Robust Default Strategy Today

For most enterprise documents, a recommended baseline pipeline is:

Raw file
   ↓
Layout and structure parsing
   ↓
Identify headings, sections, paragraphs, lists, tables, and code blocks
   ↓
Initially split at natural structural boundaries
   ↓
Recursively split oversized blocks by paragraph, sentence, or token
   ↓
Add metadata such as heading path, page number, version, and ACL
   ↓
Embedding and indexing

The priority can be summarized as:

Document structural boundaries
    >
Paragraph and sentence boundaries
    >
Fixed token boundaries
    >
Arbitrary character boundaries

Google’s layout-aware chunking ensures that content in one chunk comes from the same layout entity, such as a heading, subheading, or list, improving semantic consistency and reducing retrieval noise. Microsoft likewise recommends generating chunks from structures such as headings, paragraphs, tables, and page layout instead of relying only on fixed lengths. (Google Cloud Documentation)

3. Fixed-Length Chunking Still Has Value

Fixed-length chunking is not obsolete.

Its advantages are:

• simplicity;
• speed;
• reproducible results;
• easy control over index size;
• usefulness as an experimental baseline.

LangChain v1 still provides character-, token-, and recursive-text splitters. Its general-purpose recursive splitter works through a sequence of separators, preserving paragraphs and sentences whenever possible while still satisfying the length limit. (Docs by LangChain)

Fixed windows are suitable for:

Logs
Chat transcripts
Continuous transcriptions
Weakly structured plain text
Rapid prototypes
Establishing a comparable baseline

They are less suitable for:

Complex PDFs
Contracts and regulations
Tables
Code
Technical documents with nested headings
Multi-column layouts

Microsoft’s Document Intelligence documentation also notes that fixed-size splitting works well for data without clear semantic structure, but should not be the default for documents that require precise context and semantic understanding. (Microsoft Learn)

4. Different Document Types Need Different Strategies

1. Markdown, HTML, Wikis, and Technical Documentation

Prefer splitting by this structure:

H1
└── H2
    └── H3
        ├── Paragraph
        ├── List
        └── Code block

Recommendations:

• avoid crossing high-level heading boundaries;
• merge short paragraphs within the same section;
• recursively split sections that remain too long;
• attach the complete heading path to every chunk;
• do not truncate code blocks, lists, or blockquotes.

LangChain v1 provides specialized splitters for HTML, JSON, and code, reflecting an important principle: each data format should use a splitter that understands its structure instead of converting everything to plain text and cutting it uniformly. (Docs by LangChain)

A chunk’s indexed text might look like:

Document: Refund Management Manual
Section: Order Processing > Refund Conditions > Orders Older Than 30 Days

Orders older than 30 days are eligible only when...

The heading path restores context missing from the local body fragment.

2. PDFs, Office Documents, and Scans

For complex documents, the first step is usually not chunking but correctly parsing the layout.

The parser needs to recover:

• reading order;
• heading hierarchy;
• paragraphs;
• page numbers;
• lists;
• tables;
• images and captions;
• multi-column layout;
• footnotes, headers, and footers.

Extracting a PDF into one long string and cutting it every 500 characters can cause:

Content from left and right columns becomes interleaved
Headers become separated from table data
Section headings disappear
Headers and footers repeatedly enter the index

Google’s current layout parsing and layout-aware chunking, as well as Microsoft’s document-layout approach, all treat recovering document structure as an important stage of high-quality RAG ingestion. (Google Cloud Documentation)

Page numbers should primarily serve as:

Citation locations
Source presentation
Audit metadata

They should not be the default semantic boundary because a sentence, table, or section may span pages.

3. Contracts, Regulations, and Policy Documents

Prefer splitting by the formal logical structure:

Chapter
Article
Paragraph
Item
Definition
Appendix

Key principles:

• do not separate a primary rule from its exceptions;
• retain the parent article heading for every sub-item;
• store article numbers in metadata;
• make definition clauses linkable to the clauses that reference them;
• split an oversized article by subclauses while repeating necessary parent context.

For example, the following two sentences should not become unrelated chunks:

Employees may apply to work remotely.
Positions that handle confidential data are excluded.

Otherwise, retrieving only the first sentence may cause the system to answer incorrectly.

4. Tables

A single value from a table is usually not independently understandable.

700

It has no retrieval value.

A more useful indexed form is:

Table: Highly Skilled Professional Salary Points
Age range: 35–39
Annual salary: JPY 7 million
Points: 25

In practice:

• preserve a small table as a whole;
• split a large table by logical groups of rows;
• repeat the table name, column names, and units in every subchunk;
• preserve the original Markdown, HTML, or structured JSON;
• optionally generate a textual description, but do not use it to replace the original data completely.

5. Code

Code should be split by syntax rather than arbitrary token boundaries:

Module
Class
Function
Method
Interface
Configuration block

A code chunk should usually include:

repository
file_path
language
class_name
function_name
signature
docstring
imports

Even if a long function must be split, retain its function name, containing class, and file path.

5. What Should the Chunk Size Be?

There is no universal optimum, but you can define experimental starting points.

Short facts, FAQs, and definitions:
Approximately 200–400 tokens

General enterprise documents:
Approximately 400–800 tokens

Long rules and analytical content:
Approximately 800–1,200 tokens

These numbers are not industry standards. They are merely candidates with which to begin experiments.

One representative product default is the current automatic chunking setting in OpenAI Vector Stores:

Maximum chunk: 800 tokens
Overlap: 400 tokens

OpenAI also permits a static strategy with a user-defined chunk size and overlap. This default should be understood only as the product default for its managed File Search; it does not prove that 800/400 is optimal for every RAG system. (OpenAI Developers)

When selecting chunk size, consider:

• how much evidence a typical user question needs;
• the length of a complete semantic unit in the documents;
• the embedding model’s input limits and long-text performance;
• whether a reranker is used;
• the final context budget;
• index count, latency, and cost;
• whether many questions require evidence across paragraphs.

6. Use Overlap Carefully

Overlap prevents important information from falling exactly across a split boundary.

Mechanical use of large overlaps, however, creates:

• duplicate vectors;
• index bloat;
• multiple highly similar results;
• repeated LLM context;
• higher reranker cost;
• duplicate citations.

A pragmatic starting point is:

Fixed windows:
Approximately 10%–20% overlap

Structural chunks:
Usually no uniform overlap

Special boundaries:
Repeat the preceding sentence, heading, article number, or required definition

Note that OpenAI’s managed File Search currently uses 50% overlap in its automatic strategy. This is a default behavior of a particular managed product and should not be treated as a general industry standard for a custom-built system. (OpenAI Developers)

A cleaner approach is usually:

Add structured context instead of copying large amounts of adjacent body text.

7. Parent–Child and Multi-Granularity Retrieval

The retrieval unit does not have to be identical to the unit ultimately given to the LLM.

A two-level structure can be built:

Parent:
A complete section, article, or larger semantic unit

Child:
A small block used for embedding and precise retrieval

Query flow:

Retrieve Child
   ↓
Locate the matching fine-grained content
   ↓
Expand to the Parent or adjacent chunks when needed
   ↓
Rerank
   ↓
Send to the LLM

This approach combines:

• the retrieval precision of small blocks;
• the contextual completeness of large blocks;
• precise citations;
• composition of evidence across passages.

Do not return the entire parent document unconditionally. Expand context dynamically according to:

• parent length;
• question type;
• match location;
• whether evidence from multiple passages is required.

8. Contextual Retrieval: Add Document-Level Context to a Chunk

Anthropic’s Contextual Retrieval generates a short contextual description for every chunk before indexing, then uses:

Contextual description + original chunk

for both embedding and BM25 indexing.

For example, the original chunk is:

Revenue for this product grew by 3% year over year.

After context is added, it may become:

This passage comes from a company's financial report for the second quarter of 2023
and discusses changes in product revenue during that quarter.

Revenue for this product grew by 3% year over year.

This reduces ambiguity when a short chunk is separated from the full document. In internal experiments, Anthropic reported that combining Contextual Embeddings and Contextual BM25 reduced retrieval failure rates, with a larger improvement when reranking was added. Those figures come from the vendor’s own datasets and configuration and cannot be generalized directly to every project. (Anthropic)

The costs include:

• additional LLM calls during ingestion;
• longer indexed text;
• higher rebuild cost;
• possible errors in generated context;
• greater evaluation and debugging complexity.

It is therefore more suitable when:

Chunks frequently become ambiguous when separated from the full document
Documents use many pronouns and local references
A retrieval dataset shows that ordinary structural splitting is insufficient
The project can accept higher ingestion cost

It should not be applied to every document by default.

9. Late Chunking: Preserve Full-Text Context Before Producing Chunk Vectors

The traditional process is:

Split first
→ Embed each chunk independently

Late Chunking instead does:

First let a long-context embedding model process a longer text
→ Then aggregate each chunk's representation from the model output

The goal is for vectors of smaller chunks to retain full-document context. The original paper reported improvements on some retrieval tasks without requiring an extra textual description to be generated for every chunk. (arXiv)

It also has practical limitations:

• requires a compatible long-context embedding model;
• has a more complex implementation and inference flow;
• still requires segmentation for very long documents;
• provides inconsistent gains across retrieval tasks.

A systematic 2026 comparison found that contextualized chunking may help standard cross-corpus retrieval but can degrade within-document retrieval; simple paragraph-based or fixed-length structural approaches performed better on some tasks. (arXiv)

Late Chunking should therefore be treated as an advanced candidate rather than a new default standard.

10. Semantic and LLM-Guided Chunking

Semantic Chunking finds boundaries from changes in sentence vectors or topic transitions. LLM-guided Chunking lets a model decide what content should remain together.

Potential advantages:

• finds topic boundaries without explicit headings;
• may be more natural for long free-form text;
• can reduce mechanical cuts through semantic units.

Potential disadvantages:

• higher ingestion cost;
• potentially unstable results;
• sensitivity to the model and thresholds;
• poor reproducibility;
• no guarantee of outperforming paragraph- or structure-based splitting.

One important conclusion of a systematic study is that complex methods do not win universally. For standard cross-document retrieval, simple structural methods were highly competitive in multiple test groups, while different tasks could favor entirely different approaches. (arXiv)

The recommended progression is therefore:

Structure-aware splitting
   ↓
Recursive sentence/token splitting
   ↓
Parent–Child or context expansion
   ↓
Contextual Retrieval / Late Chunking
   ↓
Semantic or LLM-guided Chunking

Evaluation results should prove the need for every step.

11. What Data Should Each Chunk Store?

An enterprise-grade chunk should contain more than:

{
  "text": "...",
  "vector": [...]
}

At minimum, consider storing:

{
  "chunk_id": "doc-123-v4-c08",
  "document_id": "doc-123",
  "document_version": 4,
  "parent_id": "section-12",
  "content": "...",
  "title": "育儿休业制度",
  "section_path": [
    "人事制度",
    "休假制度",
    "育儿休业"
  ],
  "page_start": 8,
  "page_end": 9,
  "chunk_index": 8,
  "token_count": 542,
  "source_uri": "...",
  "tenant_id": "...",
  "allowed_roles": ["employee"],
  "content_hash": "...",
  "effective_from": "2026-04-01"
}

Microsoft’s Chunk Enrichment guidance likewise recommends enriching retrieval records with titles, summaries, keywords, entities, and other metadata to support filtering, search, and explanation of results. (Microsoft Learn)

Important fields usually include:

Document and version
Parent–child relationship
Section path
Page number
Order
ACL
Tenant
Effective date
Content hash
Parser version
Chunking-strategy version
Embedding-model version

These fields are needed to support:

• precise citations;
• permission filtering;
• reindexing;
• document updates;
• A/B testing;
• issue tracing.

12. Chunking Must Be Determined Through Evaluation

Best practice is not selecting an algorithm that sounds advanced. It is establishing reproducible experiments.

Compare at least:

A: Fixed 400 tokens
B: Fixed 800 tokens + overlap
C: Structure-aware splitting
D: Structural splitting + Parent–Child
E: Contextual Retrieval
F: Late or Semantic Chunking

Retrieval Metrics

Recall@K
Precision@K
MRR
nDCG@K
Hit Rate

Final Question-Answering Metrics

Answer Correctness
Groundedness
Citation Correctness
Completeness
Refusal accuracy

Engineering Metrics

Number of chunks
Index size
Ingestion cost
Retrieval latency
Rerank latency
LLM context tokens
Cost per successful answer

Also analyze results separately by query type:

Short factual queries
Cross-paragraph queries
Table queries
Exact-identifier queries
Multi-document queries
Unanswerable queries
Multilingual queries

Google’s RAG evaluation guidance likewise emphasizes diagnosing retrieval and generation separately instead of looking only at the final answer, because the same final error may originate from chunking, the retriever, the reranker, or the generation model. (Google Cloud)

13. A Pragmatic Production Baseline

For most enterprise knowledge bases, start with:

1. Use a layout-aware parser to recover document structure
2. Split by headings, sections, paragraphs, lists, tables, and code blocks
3. Do not cross high-level section boundaries
4. Recursively split oversized blocks by sentence and token
5. Use tokens rather than characters as the final length constraint
6. Initially test several sizes such as 400, 600, and 800 tokens
7. Use a small overlap only where needed
8. Enrich every block with heading path, page number, version, source, and ACL
9. Preserve parent_id, chunk_index, and adjacency relationships
10. Use real query sets to compare Dense, Hybrid, and Rerank performance
11. Introduce Contextual or Late Chunking only when the baseline is insufficient
12. Record the chunking strategy and version in index metadata

OpenAI’s 800-token managed default, LangChain’s recursive splitters, Google and Microsoft’s layout-aware approaches, Anthropic’s Contextual Retrieval, and Late Chunking research actually represent solutions at different levels:

OpenAI:
A usable default configuration for a managed retrieval product

LangChain:
General implementation tools and composable splitters

Google / Microsoft:
Document structure and enterprise ingestion engineering

Anthropic:
Adding full-document context to local chunks

Research papers:
Exploring new contextualized embedding and segmentation methods

They are not mutually exclusive, and none provides the one answer that applies to every system.

Conclusion

Taken together, the material above supports the following robust RAG chunking practice:

First recover and respect document structure, then split by semantic unit; constrain block size with a token ceiling, preserve context through headings, metadata, and Parent–Child relationships, and use a real evaluation dataset to determine chunk size, overlap, and advanced strategies.

Fixed-length chunking remains an important baseline. Structural chunking is usually a more reliable default for enterprise documents. Contextual Retrieval, Late Chunking, and LLM-guided Chunking offer valuable new directions, but none is an unconditional upgrade.

What should be avoided is:

See that a platform defaults to 800 tokens
→ Copy that value to every document
→ Treat PDFs, contracts, tables, and code identically
→ Skip retrieval evaluation
→ Call it a best practice

The real best practice is not one fixed parameter, but:

An explainable splitting strategy, a rebuildable index, document-type-specific processing, and continuous data-driven evaluation.