Chunking
In one sentence
Chunking is splitting documents into smaller passages before they are indexed for retrieval, and where those splits fall largely decides whether the right answer can be found at all.
Not to be confused with Reranking, or Hallucination.
Definition
Chunking is cutting a document into smaller pieces before the system indexes it for search.
Retrieval never sees the whole document. It sees the pieces, so where you cut decides what can be found.
Retrieval works on chunks, not documents. A chunk is the unit that gets embedded, the unit that comes back from a search, and the unit the model reads. That makes the split the highest-leverage decision in a RAG system, and it is also the most underestimated one, because it is where most retrieval failures start.
The tension you are trading against
- Short chunks embed precisely, because a short passage has one focused meaning. They can also strand the answer, since the passage may not carry enough context around it to make sense on its own.
- Long chunks keep the context and dilute the embedding. A passage covering several topics has a muddled semantic center, so it matches nothing strongly.
- No size is correct everywhere. It depends on how the documents are structured and what kind of question is being asked.
Strategies, roughly in order of sophistication
- Fixed-size splitting by character or token count, usually with overlap so a sentence crossing a boundary lands in both chunks. Crude, and a fair baseline.
- Recursive splitting on structural separators: paragraph breaks first, then sentences, then words. Better, and the common default.
- Structure-aware splitting that respects what the document means by its shape: sections, headings, list items, table rows. Considerably better for structured business documents.
- Semantic chunking, which cuts where a topic shifts rather than at a fixed interval.
- Contextual chunking, where each chunk carries a summary of where it sits in the parent document before it is embedded, so an isolated passage keeps the context that makes it findable.
The four failures to design against
- Split answers. The answer to a question straddles a boundary, so neither chunk matches well and neither is sufficient alone. This is the most common failure and the most damaging.
- Orphaned references. A chunk reads "this does not apply to the premium tier" with nothing anywhere in it to say what "this" was.
- Destroyed tables. A split between rows and headers turns the numbers into noise. Business documents are full of tables.
- Lost headers. A price detached from the heading that said which product it belonged to. This one is actively dangerous, because a confidently retrieved wrong price is worse than no answer.
Common misconception
That chunking is a preprocessing detail you configure once and forget. It is the single most impactful tunable in a RAG system. Teams routinely swap the embedding model or the language model chasing quality that a better boundary would have delivered for less.
Why it matters commercially
When chunking fails it produces confidently wrong answers about prices, policies and eligibility, which are precisely the answers a business cannot afford to get wrong. Lost headers in particular are a mechanism for quoting the wrong price with full confidence.
In voice specifically
Spoken answers have to be short. A chunking strategy that produces large chunks forces either a long spoken answer or an extra summarization step, and both cost time in a path where time is the binding constraint.
Where AsqVox fits
Chunking quality decides how well the Orb's true RAG performs over documents a business uploads. Those documents lean heavily on tables, headings and tiered pricing, which are precisely the structures naive chunking handles worst.
Visual
The same document, split three ways
Most teams change the model. The boundary was the problem.
Statistics
Every figure carries its source and year. Vendor numbers are labelled as vendor numbers, and where no reliable figure exists this page says so rather than borrowing one.
No reliable published benchmark establishes an optimal chunk size. Practice is empirical and corpus-specific.
-no reliable figureAny specific token count in circulation should be treated as a starting point for testing rather than as a finding. This is the honest framing the page leads with, and it is worth stating before any strategy comparison.
No cross-industry figure exists for retrieval accuracy by chunking strategy. Nobody has measured how much better structure-aware chunking is than fixed-size chunking across corpora.
-no reliable figureThe strategies can be ranked by argument and by what they preserve. They cannot be ranked by a published number, and a vendor offering you one is quoting its own corpus.
Chunk overlap is standard practice, adopted specifically to mitigate answers that split across a boundary.
Standard practiceindustry rangeEngineering consensus, 2026 - Consensus on the practice, not on the amount. How much overlap to use is another empirical question with no published optimum.
Structure-aware and contextual chunking are the direction of practice for business documents, because those documents carry meaning in their structure rather than only in their prose.
Direction of practiceindustry rangeEngineering consensus, 2026 - A statement about where practice is heading, not a measured result. It is the reason table handling is worth asking a vendor about directly.
Chunk size drives how much text reaches the model, which drives token count, which drives both cost and time to first token. Against a sub-second time to first audio target, oversized chunks carry a measurable product cost.
Under 1,000 msindustry rangeIndustry working threshold, 2026 - A working convention rather than a published standard. The point for chunking is that the boundary decision is a latency decision as well as a quality one.
Real-world word error rates of roughly 8 to 12 percent for leading recognizers mean the query arriving at retrieval may already be imperfect.
8 to 12%industry rangeReported real-world word error rate for leading recognizers, 2026 - The compounding problem in voice. A chunking strategy that depends on precise query matching is fragile before retrieval has even started, because the query it receives is not necessarily the sentence the visitor said.
Examples
In practice
A software company's voice agent quotes the wrong price roughly one time in eight. The pricing table turns out to be split so that tier names sit in one chunk and the figures sit in the next. Retrieval returns the figures chunk, which holds three numbers and no product names, and the model attributes them by guessing. Re-chunking so each table row stays intact with its header prefixed removes the failure entirely. The model was never touched.
The everyday version
Chunking is the step where your documents get sliced into pieces before the system files them away. Slice in the wrong spot and a price winds up separated from the product name it belongs to. The system then retrieves the price, has no way to tell what it is for, and guesses with full confidence.
Usage
Who says it
- AI and search engineers, constantly. It is usually the first thing they tune when retrieval underperforms.
- It rarely comes up in a buyer conversation, though it should appear more often in vendor explanations of why their retrieval works.
Where it turns up
- Next to supported document formats, ingestion behavior, handling of tables and structured content, and whether chunking is configurable at all.
- Table handling is the question worth asking directly. Business documents are table-heavy and naive ingestion destroys them.
Common misuse
- Taking a chunk size from a tutorial and treating it as a universal setting.
- Blaming the language model for retrieval failures that started at a chunk boundary. This misdiagnosis is extremely common.
- Skipping tables and structured content during ingestion testing, then discovering the problem when a customer has been quoted the wrong figure.
Questions people ask
What is the best chunk size for RAG?
No published benchmark establishes one. Practice is empirical and corpus-specific, so any token count you see recommended is a starting point for testing rather than a finding. Short chunks embed precisely and can strand the answer. Long chunks keep the context and blur the embedding. Test against your own documents and the questions people actually ask.
Why does my RAG system quote the wrong price?
Usually a chunk boundary rather than the model. When a pricing table is split so the tier names land in one chunk and the figures in the next, retrieval returns three numbers with no product attached and the model guesses which belongs to which. Keeping each table row intact with its header prefixed fixes it without any model change.
What is chunk overlap and why is it used?
Overlap means consecutive chunks share some text at their edges, so a sentence that crosses a boundary appears in both. It exists to mitigate split answers, where the answer straddles two chunks and neither matches well alone. Overlap is engineering consensus as a practice, but there is no published optimum for how much of it to use.
Does chunking matter more for voice than for text?
Yes, in two ways. Spoken answers have to be short, so large chunks force either a long answer or an extra summarization step, and both spend time inside a sub-second response budget. And speech recognition arrives with real-world word error rates of roughly 8 to 12 percent, so a chunking strategy that needs precise query matching is fragile before retrieval starts.
Last reviewed 30 July 2026. Written and reviewed by Dhruv Dholakia, founder of AsqVox.