Chunking Is a Product Decision: How Document Splitting Shapes Answers

Chunking decides the atomic unit of knowledge a retrieval system can ever return — not how well the model reasons, but what it's even allowed to see. A retriever hands back whole chunks only, so evidence split across a chunk boundary can go permanently unretrievable, no matter how good the model downstream is.

Quick answer: Chunking — how a document gets split into retrievable pieces — determines what a RAG system can find, not just how fast it finds it. Get chunk size, overlap, or split logic wrong and a correct answer sitting right there in the source becomes unretrievable, no matter how good the model downstream is.

What Chunking Actually Decides in a RAG System

Chunking decides the atomic unit of knowledge a retrieval system can ever return. It sets a hard ceiling on retrievability before the retriever, reranker, or model ever runs — if the fact needed to answer a question is split across two chunks, the system has to get lucky and retrieve both, or it simply cannot answer correctly.

Most RAG debugging starts at the wrong layer. Teams tune prompts, swap embedding models, or upgrade to a bigger LLM when the real defect is upstream: the source document was split so that the fact needed to answer a question never exists inside one retrievable unit. This is a context engineering problem, not a prompting problem — a distinction worth understanding before debugging a retrieval failure as if it were a wording issue, which is exactly why the difference between context and prompt engineering matters here.

For a PM, chunking is best understood as an information-architecture decision — structurally identical to deciding how content gets organized into pages, sections, and cards in a product. Cut a source document into pieces that ignore its logical structure, and the system's memory of that document is fragmented for good. No amount of retrieval tuning downstream fixes a chunk that was born wrong, which is one reason a complete guide to context engineering treats chunking as a first-class design decision alongside retrieval, ranking, and packet assembly, not an implementation detail.

Three variables define any chunking strategy, and a PM reviewing a RAG build should be able to name the trade-off in each:

• Chunk size — how much text becomes one retrievable unit.
• Overlap — how much adjacent chunks repeat, to soften boundary loss.
• Split logic — whether cuts happen at a fixed character count, or respect the document's actual structure or meaning.

Get any one wrong on a document your product depends on, and the failure looks like "the AI got it wrong" when the AI never had a chance.

A Policy PDF, Two Chunking Strategies: One Right Answer, One Wrong One

A 40-page travel-and-expense policy with a per-diem rate table shows the failure mode precisely. Fixed-size chunking that ignores table structure can separate a rate from the column header that gives it meaning, so the retriever returns an orphaned number and the model either guesses or answers with the wrong duration bracket.

The source document

Suppose page 14 of the policy contains this table, with a caption above it:

Table 6: Meal per diem by trip duration

A traveler asks the company's policy assistant: "What's the international meal per diem for a trip of 14 days or more?" The correct answer is in the table, unambiguously. Whether the assistant can find it depends entirely on how page 14 was chunked months earlier, at ingestion time.

Where fixed-size splitting breaks

A naive pipeline splitting every 300 characters, with no awareness of table structure, might cut the page like this:

Chunk A: "...Table 6: Meal per diem by trip duration
| Travel Duration | Domestic Meal Per Diem | International Meal Per Diem |
| 1-3 days | $55 | $71 |
| 4-13 days | $65 | $85 |"

Chunk B: "| 14+ days | $75 | $99 |

All receipts over $25 require an itemized..."

The row with the answer — $99 — lives in Chunk B. The header row that defines which column is domestic and which is international lives in Chunk A. If the retriever's top match for "14+ days international per diem" is Chunk B alone, the model sees two bare numbers with no labels and either guesses, defaults to the more common domestic figure, or declines to answer. The document had the right answer. The chunking made it unreadable.

What structure-aware splitting preserves

A splitter that treats tables as atomic units — recognizing "this is one table, don't cut inside it" — keeps the caption, header row, and every data row together in a single chunk, or repeats the header row via overlap in every chunk that contains part of the table. Retrieval returns a self-contained chunk; the model reads $99, matches it to "International" and "14+ days," and answers correctly with a citation back to Table 6.

The lesson generalizes past tables: the same failure happens when a chunk boundary falls between a policy clause and its exception, between a procedure step and the condition that changes it, or between a defined term and its definition. Anywhere a document pairs a fact with the label or qualifier that makes it meaningful, a careless split can separate them.

Chunk Size and Overlap: The Trade-offs Behind the Numbers

Chunk size trades precision against context. Small chunks (roughly 100-300 tokens) isolate a fact precisely but strip away the surrounding qualifier that changes its meaning; large chunks (800-1,500+ tokens) preserve context but dilute the embedding's relevance signal and cost more tokens per retrieved item.

There is no size that wins everywhere — it's a bet on what kind of question the document will be asked most often. A support-macro knowledge base leans small; a legal-review assistant summarizing whole clauses leans large.

Why overlap exists, and how it can backfire

Overlap duplicates the tail of one chunk at the head of the next, so a fact sitting near a boundary usually survives in at least one chunk intact. Guides from LangChain and Pinecone commonly recommend overlap in the 10-20% range of chunk size as a starting point, tuned from there against real queries.

Too little overlap reproduces exactly the per-diem failure above — a fact stranded on the wrong side of a cut. Too much overlap creates near-duplicate chunks that compete for the same top-k retrieval slots, inflating token cost in the eventual context packet without adding new information, and can even crowd out a genuinely different, relevant chunk from making the cut.

Anthropic's own published research on contextual retrieval (2024) tackled this same boundary-loss problem directly: prepending a short, chunk-specific summary of where a passage sits in the source document — before embedding it — cut failed-retrieval rates by roughly a third on its own, and by roughly two-thirds once combined with reranking. The fix wasn't a bigger model; it was giving each chunk back the context that splitting had removed.

Fixed-Size vs. Semantic vs. Structure-Aware Splitting: Choosing How to Cut

Three families of split logic exist, and they disagree on what should determine a cut point: a raw character count, a shift in meaning between sentences, or the document's own formatting. Which one fits depends on how structured the source document already is.

This taxonomy roughly follows Greg Kamradt's widely referenced "5 Levels of Text Splitting" framework, which orders these approaches from crude to sophisticated and is worth knowing by name when a RAG engineer says "we're just using recursive character splitting" — that sentence is a strategy decision, not a technical footnote.

Structure-aware splitting depends on the pipeline actually recognizing structure in the first place. Document-parsing libraries like Unstructured.io classify PDF content into titles, list items, and tables before chunking happens, which is what makes "don't cut inside this table" an enforceable rule rather than a hope. Without that layer, a splitter is guessing at structure from whitespace and font size, which is far less reliable on real-world scanned or exported PDFs.

Heuristics for Choosing a Chunking Strategy

The working heuristic is simple to state and easy to skip under deadline pressure: match chunk boundaries to the atomic unit of the domain's real facts — a table row, a policy clause, a procedure step, a FAQ answer — rather than an arbitrary character count, then test the result against real questions before trusting a default.

1. Start from the atomic fact, not the token budget. Identify the smallest unit that has to survive intact — a per-diem row, a contract clause, a defined term with its definition — and size chunks so that unit never gets cut mid-way.
2. Respect native structure before imposing artificial size. If the source already has headings, tables, and lists, a structure-aware splitter should use them as boundaries first; a flat character count is a fallback for unstructured text, not a default for everything.
3. Set overlap proportional to sentence or list-item length, not a flat number. A 200-character overlap does nothing for a table row that's 400 characters wide; it should cover at least one full logical unit at the boundary.
4. Map chunking to the job each query is doing, not just the average query. A single policy document gets consulted for very different reasons — a quick lookup, a full read during onboarding, a detailed defense during an audit — and the Jobs-to-Be-Done framework is a useful lens for naming which job a given chunk actually needs to serve before sizing it.
5. Think across the document's touchpoints, not just its content. The same per-diem table gets hit at onboarding, at expense-filing time, and during a dispute — different moments implying different granularity of question, the same way mapping a customer journey surfaces that a single artifact serves very different needs at different stages.
6. Decide what context a chunk must carry before deciding its size. This is the same question posed in what to include in a context window: a chunk boundary is really a decision about which surrounding context is mandatory for a fact to stay interpretable outside its original document.
7. Build a small eval set of real questions and measure recall before shipping a default. Ten to twenty representative questions, run against the actual chunked index, will surface a bad boundary decision faster than any amount of theorizing about ideal chunk size.

Signs Your Chunking Is Losing Answers

The clearest sign of a chunking problem is a correct answer that exists verbatim in the source document but never appears in retrieval results, no matter how the query is phrased. That single symptom — evidence present, retrieval empty — is close to diagnostic on its own, because it rules out the model as the cause.

Watch for these related patterns once a RAG system is live:

• Retrieved chunks contain partial numbers, table fragments, or dangling references — a figure with no label, a "see above" with nothing above it. This is the per-diem failure showing up in production logs.
• Raising top-k "fixes" wrong answers. If pulling in 10 chunks instead of 4 suddenly gets the right answer, the real defect is that chunks are too small or badly split, and the system is compensating by brute-forcing enough overlap to catch the fragments by luck.
• The same question, phrased two different ways, gets contradictory answers. This usually means the relevant fact is split across chunks that different phrasings happen to retrieve differently — a symptom of a boundary problem, not a model inconsistency problem.
• Citations point to the wrong section of a long document, even when the final answer sounds plausible. The model is often synthesizing from a chunk adjacent to the right one, not the one that actually contains the fact.
• Answers degrade specifically on long or heavily tabular documents while short documents perform fine — a sign the chunking strategy was tuned on documents that never stressed the size and overlap settings being used everywhere.

Making the split a documented decision, not a hidden default

Most RAG stacks bury this decision inside an ingestion script's chunk_size and chunk_overlap arguments — invisible to anyone who isn't reading pipeline code, and easy to forget was ever a choice at all. That's a governance problem as much as a technical one.

Prodinja's own approach, as an in-progress prototype, treats retrieval as an explicit packet decision: how a source is prepared and split is meant to be a documented, inspectable choice, not a hidden default buried three files deep. A PM reviewing why an answer came back wrong can actually see how the source was cut, not just what the model said in response.

Key Takeaways

• Chunking sets a hard ceiling on retrievability before the retriever, reranker, or model ever runs — a fact split across a chunk boundary may never be found, regardless of downstream quality.
• Chunk size trades precision for context. Small chunks (100-300 tokens) isolate facts but lose qualifiers; large chunks (800+ tokens) keep context but dilute relevance ranking and raise token cost.
• Overlap is a hedge against boundary loss, typically 10-20% of chunk size as a starting point — too little strands facts at cut points, too much creates redundant chunks competing for top-k slots.
• Structure-aware and semantic splitting beat fixed-size splitting on documents with real formatting — tables, clauses, numbered steps — because they cut where the document already has a boundary, not where a character counter runs out.
• Pick a strategy from the domain's atomic fact unit, not a generic default, and validate it against a real question set before shipping — ten to twenty representative queries reveal a bad boundary faster than any amount of theorizing.
• A correct answer that exists in the source but never surfaces in retrieval is the clearest diagnostic signal of a chunking defect, distinguishable from a model-quality problem.
• Treat chunking as a visible, documented decision rather than a buried ingestion default, so it can be reviewed and debugged like any other product choice.