A split table returns wrong numbers with total confidence; naive chunking treats a financial table, a code block, and a two-column PDF page as if they were the same undifferentiated prose your fixed-size splitter was built for. Parsing for retrieval means detecting structure before you chunk — tables, code, and layout each need their own extraction logic, or the retrieved "context" is just shrapnel with a page number attached.
Quick Answer: Fixed-size or fixed-token chunking breaks structured content (tables, code, multi-column PDFs) because it splits on character count, not semantic boundaries. Structure-aware parsing — detecting tables via layout models, keeping code blocks atomic, and reading PDFs in reading order — prevents a retrieved chunk from containing half a row, half a function, or two unrelated columns stitched together.
Why naive chunking is actively hostile to structured content
Fixed-size chunking (say, 512 tokens with 50-token overlap) was designed for prose, where meaning survives being cut at an arbitrary sentence boundary. It answers "how much text fits in a chunk," never "does this chunk contain a complete idea." Tables, code, and multi-column layouts all violate that assumption in different ways.
A table's meaning lives in the relationship between a row and its header — sever that relationship and every number becomes unlabeled. A code block's meaning lives in matching braces, function scope, and surrounding comments — cut mid-function and you retrieve a fragment that won't even parse. A multi-column PDF's meaning depends on reading order — extract text in raw byte order and column A's ending sentence gets glued to column B's opening sentence.
- Tables: split rows lose their header row context; a retrieved fragment is a set of numbers with no label
- Code: split functions lose closing braces, imports, or the docstring explaining intent
- Multi-column PDFs: raw text extraction interleaves columns, producing sentences that never existed in the source
- Footnotes and callouts: often extracted inline, mid-sentence, breaking the paragraph they were annotating
Our companion piece on chunking strategy and retrieval quality covers this failure mode for prose generally. Structured content is the sharper case: prose degrades gracefully when split badly — a paragraph cut in half is still mostly readable. A table cut in half is often actively misleading, because the remaining fragment still looks complete.
Before/after: a split financial table producing a wrong answer
A financial table split by a naive chunker is worse than no table at all — the model still answers, just with a confident number attached to the wrong row. This is the clearest demonstration of why structure-aware parsing isn't a nice-to-have for finance, ops, and compliance documents.
The source table
Imagine a 10-K excerpt with quarterly revenue by segment:
| Segment | Q1 Revenue | Q2 Revenue | Q3 Revenue | Q4 Revenue |
|---|---|---|---|---|
| Hardware | $42.1M | $44.8M | $39.2M | $51.6M |
| Software | $118.3M | $121.9M | $126.4M | $134.7M |
| Services | $28.7M | $30.1M | $31.9M | $33.2M |
A fixed-size chunker operating on the extracted text (tables are usually flattened to plain text before chunking unless you intervene) might cut at the 512-token boundary mid-table. The result:
Chunk 1 (retrieved) contains: Segment | Q1 Revenue | Q2 Revenue\nHardware | $42.1M | $44.8M\nSoftware | $118.3M | $121.9M\nServices | $28.7M | $30.1M
Chunk 2 (not retrieved, ranked lower by the embedding similarity search) contains: Q3 Revenue | Q4 Revenue\n$39.2M | $51.6M\n$126.4M | $134.7M\n$31.9M | $33.2M
The question: "What was total Services revenue for the year?"
What happens: The retriever pulls Chunk 1 because it embeds closer to "Services revenue." The model, working only from Chunk 1, sees Q1 and Q2 Services figures ($28.7M + $30.1M) and either reports $58.8M as if it were the full-year figure, or fabricates a plausible-sounding annual estimate. The correct answer — $123.9M across all four quarters — never enters the model's context because Q3 and Q4 live in a chunk that wasn't retrieved.
This is not a hallucination in the usual sense. The model is reasoning correctly over incomplete data it was never told was incomplete. The failure is entirely upstream, in parsing and chunking, and no amount of prompt engineering fixes it after the fact.
| Approach | What's retrieved | Answer produced |
|---|---|---|
| Fixed-size chunking (table flattened, split mid-row group) | Q1-Q2 only, no header repetition on Chunk 2 | Wrong: partial-year figure presented as annual |
| Structure-aware parsing (table kept atomic, retrieved as one unit) | Full table, all four quarters, header intact | Correct: $123.9M, traceable to source row |
The fix at the parsing layer: detect the table as a single semantic unit before chunking begins, and either keep it whole (if it fits the model's context budget) or split it in a way that repeats the header and segment label in every fragment — never splits across the numeric columns of a single row group.
Structure-aware parsing: detect before you split
Structure-aware parsing means running a layout or structure detection pass on the document before any chunking decision, so the chunker receives labeled blocks (table, code, heading, prose) instead of an undifferentiated character stream. This single ordering change — detect, then chunk — is the difference between retrieval that works and retrieval that looks like it works until someone checks the numbers.
1. Layout detection first. Tools like unstructured, LlamaParse, and Amazon Textract's table-extraction API run a layout model over the raw PDF or document to classify regions before text extraction begins. This is fundamentally different from pdfplumber or raw PyPDF2 text extraction, which pulls characters in stream order with no notion of "this is a table."
2. Tables become their own object, not flattened text. A properly parsed table should be extractable as structured data (JSON rows, a DataFrame, or markdown with header preserved) — not a wall of pipe-separated text that a downstream token-count chunker can still slice through.
3. Code blocks respect syntactic boundaries. Chunk code by function or class definition (using an AST parser or even a simple bracket-matching heuristic), never by token count. A retrieved fragment should be a complete, parseable unit — ideally with its docstring or leading comment attached, since that's often where intent lives.
4. Every table/code chunk carries a caption or label. Repeat the table's title, the preceding section heading, or the function's file path in the metadata (or even the text) of every chunk derived from it — so a retrieved fragment is self-describing even out of context.
Structure-aware parsing doesn't guarantee a table always stays in one chunk — a 40-row table may still legitimately need splitting for a small context window. It guarantees the split, when it happens, happens along row-group boundaries with headers repeated, not at an arbitrary character count.
This is one of the four product decisions RAG forces on a technical PM early: parsing strategy isn't a backend implementation detail you can defer — it determines whether the retrieval layer is trustworthy for the exact document types (financial tables, technical specs, contracts) that usually carry the highest cost of being wrong.
Layout-aware extraction for multi-column and mixed-media PDFs
Multi-column PDFs require reading-order detection, not raw text-stream extraction, because a two-column academic paper or investor deck interleaves unrelated sentences the moment you read left-to-right across the page instead of down each column first. This is a distinct problem from table-splitting, but it produces the same symptom: retrieved chunks that are grammatically plausible and factually scrambled.
Common failure sources in real PDF pipelines:
- Multi-column layouts: raw extraction (PDF text streams,
pdftotext -layoutdisabled) often reads left-to-right across the full page width, splicing column 1's last line to column 2's first line - Headers/footers repeated on every page: injected into the middle of extracted body text unless explicitly filtered by position
- Embedded images and charts: text extraction skips them entirely, silently dropping information a human reader would treat as load-bearing (a trend chart referenced in the surrounding prose)
- Scanned PDFs (image-only): require OCR (Tesseract, AWS Textract, Google Document AI) before any text-based parsing can happen at all — a step teams frequently discover is missing only after retrieval quality is already bad in production
A layout-aware extraction pipeline typically does the following, roughly in order:
- Detect page layout regions (columns, headers, footers, tables, figures) with a layout model
- Extract text per region, respecting each region's internal reading order
- Reassemble regions in the human reading order (top-to-bottom, left column then right column) rather than raw stream order
- Tag tables and figures as separate structured objects rather than merging them into surrounding prose
- Run OCR as a fallback only where the region is image-only, not across the whole document indiscriminately
A note on Docling and open-source layout models
IBM's open-source Docling project and the broader research it draws on (layout models trained on datasets like PubLayNet and DocBank) represent the general direction the field has moved: treat a document as a set of typed regions before treating it as a stream of characters. A technical PM evaluating a parsing vendor or library should ask specifically whether it does region classification before extraction, or extraction alone — the two produce very different downstream retrieval behavior.
Deciding what a usable retrieved unit looks like for non-prose content
The hardest parsing decision isn't detection — most modern layout tools detect tables and code reasonably well — it's deciding, per document type, what a complete retrievable unit actually is. A table might need to be one unit; a long table might need to be split by logical row groups; a code file might need per-function units with shared file-level context.
This is a product decision as much as an engineering one, because it trades off recall (retrieve broadly, risk noise) against precision (retrieve narrowly, risk missing context), and the right tradeoff differs by document type and by the cost of a wrong answer. A support-macro knowledge base can tolerate looser units; a financial or compliance corpus generally can't.
Key Takeaways
- Fixed-size chunking splits on character count, not structure — it has no concept of "this is a table row" or "this is a function body," so it cuts wherever the token budget runs out.
- A split table is worse than no table: the retrieved fragment still looks complete, so the model answers confidently from partial data rather than declining to answer.
- Detect structure before you chunk — layout-aware tools (
unstructured,LlamaParse, Textract,Docling) classify tables, code, and columns before extraction, not after. - Code should chunk on syntactic boundaries (function/class, via AST or bracket matching), never on token count, so retrieved fragments remain complete and parseable.
- Multi-column PDFs need reading-order reconstruction, not raw text-stream extraction, or unrelated columns get spliced into scrambled sentences.
- Repeat labels and headers in every derived chunk — a table split across fragments should carry its header and section title in each piece, so no fragment is retrievable without its own context.
- Choosing the retrievable unit is a product decision, not just an engineering one — it trades recall against precision differently for a support wiki than for a financial 10-K.
Frequently Asked Questions
What's the best way to parse PDFs for RAG?
The best approach runs layout detection before text extraction, classifying regions as tables, code, headers, or prose so each can be chunked with logic appropriate to its structure. Tools like unstructured, LlamaParse, and Amazon Textract do this; raw pdftotext-style extraction does not, and will scramble multi-column or table-heavy documents.
How do you chunk tables without breaking them?
Extract the table as a structured object (rows and headers, not flattened text) before chunking begins, then either keep the whole table as one retrievable unit or split by row group while repeating the header and table title in every resulting fragment. Never let a generic fixed-size chunker slice through a table's flattened text — see the chunking strategy guide for the general splitting principles this builds on.
Why does my RAG system give wrong answers from tables?
Wrong answers from tables usually mean the table was flattened to plain text and split by a token-count chunker, so the retrieved fragment contains some rows but not others — often missing the header that would label the numbers correctly. The model then answers from a plausible-looking but incomplete slice of the table, as shown in the financial-table example above.
Do I need OCR for PDF parsing?
Only for image-only or scanned pages — if the PDF has an underlying text layer, OCR is unnecessary and can introduce its own errors. Run OCR (Tesseract, AWS Textract, Google Document AI) selectively on regions your layout detector flags as image-only, rather than across an entire mixed-content document.
How does document parsing connect to the rest of a RAG strategy?
Parsing decisions constrain everything downstream — chunking, retrieval quality, and even permission enforcement, since a mis-parsed document can leak content across the boundaries covered in access control and permissions for RAG. For the full picture of how parsing, chunking, retrieval, and generation fit together, see the complete guide to RAG and knowledge retrieval, and for the discovery work that determines which documents even need this level of care, see the complete guide to jobs-to-be-done and how it surfaces which workflows depend on getting a specific number right, the same way the customer journey framework surfaces where a wrong answer actually costs the user something.