Chunking and Structural Context: How Document Boundaries Change Retrieval

A retrieval system can only return what a chunk contains. Cut the answer in half, and no reranker will put it back together.

You have a working RAG pipeline. Embeddings are reasonable, the vector store is fast, and the generator is competent. Yet some queries come back with a chunk that contains every word of the question and none of the meaning. The answer was there in the source document—just not in the window your splitter happened to carve out.

That symptom is rarely an embedding problem. It is a boundary problem. And it persists because most teams still carry a weak mental model of chunking: that it is a size parameter you set once and forget. The stronger model is this: chunking is a boundary decision that determines what structural context survives into retrieval. The chunk is the atomic unit your retriever can score and return. Whatever the boundary excludes, the generator never sees.

This article is a sequence of experiments, not a list of strategies to memorize. We will treat chunk boundaries, metadata, overlap, and hierarchy as variables you can isolate and measure on your own corpus.

Why Chunk Boundaries Decide Retrieval Quality

Start with the mechanism. Retrieval scores chunks, not documents. A boundary error is therefore unrecoverable downstream: if the sentence that answers the query landed in the neighboring chunk, the generator receives a confident, well-formed, incomplete context. It will answer anyway. That is the trap.

Fixed token windows are a useful baseline precisely because they are structure-blind. They ignore headings, tables, and numbered procedures, which makes them the control condition against which structural gains become measurable rather than assumed. If you never run the dumb baseline, you cannot prove your clever splitter earned its complexity.

Small corpora hide boundary damage. With a few hundred documents, near-neighbor noise is low, semantic similarity stays tightly clustered, and weak chunking still looks accurate. Scale exposes it. Chunk count grows faster than document count, the vector space gets dense and noisy, and "nearest neighbors" stop being genuinely relevant. The failure was always there; you just could not see it yet.

Name the tradeoff axis now, because every later decision sits on it: precision versus context completeness. Small, focused chunks retrieve sharply but arrive meaning-starved. Large chunks carry their own meaning but blur the embedding and crowd the top-k. Chunking is the act of choosing where on that axis each piece of your corpus should live.

Read the Document Before You Split It

The right chunker depends on what structure your sources actually have. Inventory that first.

Structure level	Examples	Typical approach
Strongly structured	Forms, records, invoices	Prebuilt or parser-driven extraction
Semi-structured	Web pages, Markdown, HTML	Layout-aware parsing, heading splits
Inferred	Regulations, prose reports	Boundary- or sentence-based splitting
Unstructured	Notes, transcripts, chat logs	Sentence-based with overlap

This spectrum is not academic. A form has fields you can extract directly; a regulation has section numbers you must infer from prose. Applying a Markdown splitter to a scanned PDF, or a fixed window to a form, produces garbage at a predictable rate.

Conversion is part of chunking, not a step before it. PDF-to-Markdown extraction that scrambles tables or drops headings poisons every downstream chunk. A table whose header row is separated from its data rows is worse than no table at all, because it looks like clean text. Inspect the parsed output before you tune a single splitter parameter.

If a human cannot tell where a section starts by reading the parsed text, no splitter will infer it. Fix the parse first.

The practical check is unglamorous: dump the parsed text and read it. Ten minutes of reading beats an afternoon of parameter sweeps.

Chunking Strategies and What Each One Preserves

Compare strategies on one axis: what structural information stays intact.

Fixed-size and sliding-window. Fast, predictable, structure-blind. They fragment tables, procedures, and cross-references. Keep this as your control.

Sentence and paragraph. Respect linguistic units but ignore hierarchy, and they break on complex layouts where a "paragraph" spans a page boundary.

Recursive splitting. Tries the highest-level delimiter first—paragraph, then sentence, then word—and falls back only when the size target forces it. This gives you structural awareness with size control, which makes it a strong baseline for mixed text when native structure is incomplete or inconsistent.

Semantic chunking. Places boundaries where embedding similarity shifts. Useful when structure is genuinely absent, but its behavior depends entirely on the representation supplied to it: if your parser drops headings, tables, or layout cues, the embedding pass never sees them. It also costs an embedding pass during ingestion.

Structure-native splitting. Splits on headings, sections, and list items. Keeps numbered steps, list runs, and table rows with their headers. This is the strategy that preserves the most meaning when your source structure is trustworthy.

One rule cuts across all of them: no strategy wins universally. Treat the choice as a hypothesis to test on your corpus, not a best practice to adopt. I have watched teams adopt semantic chunking because it sounded sophisticated, then quietly lose to a recursive splitter on their own evaluation set.

Boundary Repair Happens at Three Different Stages

The draft of this article originally grouped metadata, context prefixes, hierarchical indexing, and parent/neighbor expansion under one heading. That was a mistake worth naming, because it is the mistake most teams make: treating every context-restoration technique as interchangeable. They are not. They operate at different pipeline stages, and each one repairs a different failure.

Representation-time enrichment changes the text that gets embedded and searched. A context prefix—a short, chunk-specific line describing where the chunk sits in the document—belongs here. It restores meaning lost at the boundary before the embedding is computed. This is a low-cost intervention with an outsized effect on retrieval failure rates, and it composes well with metadata.

Candidate-selection metadata travels with the chunk but does not change its embedded text. Source, position, section path, and document title belong here. A chunk that knows it came from "Section 4.2: Refund Eligibility" is interpretable even if the heading text itself was cut. Metadata also supports filtering before retrieval—essential when validity windows or access control matter, because filtering after retrieval wastes top-k slots on candidates you will discard.

Post-retrieval context expansion changes what the generator sees, not what the retriever scored. Hierarchical indexing lets retrieval operate at document, section, and chunk level; a precise hit can then be expanded with its parent or neighbors after the fact. Retrieve narrow, reconstruct wide.

Overlap sits awkwardly across all three. It reduces boundary loss by duplicating text across adjacent chunks, but the cost is real: duplicated embeddings, inflated index size, and—worse—near-duplicate chunks consuming multiple top-k slots. If your query matches the overlap region, both neighbors score highly and you have spent two retrieval slots on one piece of evidence.

Decision rule: when storage and top-k budget matter, prefer metadata and hierarchy prefixes over large overlap. Overlap buys boundary coverage with redundancy; metadata buys it with structure.

A Worked Trace: One Boundary, Three Repairs

Abstract taxonomy is not enough. Here is a compact trace. Suppose your source document contains this section:

## 4.2 Refund Eligibility

Refunds are available within 30 days of purchase.
Digital goods are excluded unless the license
was never activated. Activation is defined in
Section 7.1.

The query is: "Can I get a refund on a digital license I never activated?"

Fixed-window chunking (256 tokens, no overlap). The window boundary lands after "Digital goods are excluded unless the license." The next chunk begins with "was never activated. Activation is defined in Section 7.1." The retriever scores the first chunk highly—it contains "refund," "digital goods," and "excluded." The generator receives a chunk that says digital goods are excluded, with no exception clause. It answers: no. Wrong.

Structure-first chunking. The whole section is one chunk. The retriever returns it intact. The generator sees the exception and answers: yes, if the license was never activated. Correct.

Fixed-window plus hierarchy prefix. The first chunk is prepended with "Section 4.2 Refund Eligibility —". This does not recover the missing exception clause, but it does tell the retriever and generator that this chunk is about refund eligibility, not general digital-goods policy. Evidence-span recall improves for queries about the section topic; it does not improve for queries that need the exception.

Fixed-window plus parent reconstruction. The retriever still scores the first chunk, but the pipeline expands it to its parent section before generation. The generator now sees the full section, including the exception. Correct—at the cost of a larger context window and a parent-index lookup.

The trace shows the boundary between repairs. Structure-first fixes the boundary. A prefix fixes interpretation. Parent reconstruction fixes completeness after the fact. Only one of them addresses the actual failure in this example.

Design a Chunking Experiment You Can Trust

The original experiment design in this article changed too many things at once. It varied chunk boundaries, retrieval text, reconstruction, and indexing level across adjacent conditions, which means a gain could not be attributed to any single variable. That defeats the purpose. Here is a staged design that isolates each factor.

Step 1: Build a small labeled set. Real queries paired with the exact evidence spans that should be retrieved. Fifty well-chosen pairs beat five hundred synthetic ones.

Step 2: Stage A — isolate boundary strategy. Hold retrieval method (dense cosine), top-k, and post-retrieval context constant. Vary only the splitter.

A1. fixed token windows          (control)
A2. recursive splitting
A3. structure-first splitting

Index each variant separately. Return the same number of chunks to the generator. No prefixes, no parent expansion, no reranking. Measure which boundary strategy retrieves the gold evidence span most often.

Step 3: Stage B — isolate representation enrichment. Take the winning boundary strategy from Stage A and hold it fixed. Vary only what text gets embedded.

B1. chunk text only              (control)
B2. chunk text + hierarchy prefix
B3. chunk text + document title

Same retrieval method, same top-k, same generator context. If B2 beats B1, the gain is attributable to the prefix, not to a boundary change.

Step 4: Stage C — isolate post-retrieval reconstruction. Take the winning configuration from Stage B and hold it fixed. Vary only what the generator receives.

C1. retrieved chunk only         (control)
C2. chunk + parent section
C3. chunk + neighboring chunks

Same retriever, same candidates, different context assembly. Measure end-to-end answer accuracy and context tokens consumed.

Step 5: Evaluate retrieval before generation. Track recall@k, MRR or nDCG, evidence-span recall, index size, latency, and context tokens passed to the generator. Evidence-span recall—did the retrieved chunk actually contain the gold span—is the metric that exposes boundary loss most directly.

Step 6: Split failures by type. Boundary loss, missing context, near-duplicate crowding, and wrong-section match each point at a different fix. A single aggregate score hides which one you have.

Step 7: Keep the control. If fixed windows win on your corpus, that is a result, not a failure to be sophisticated. Report it and move on.

Failure Modes and When Structure-First Backfires

Structure-first chunking is not free. It fails in predictable ways.

Untrustworthy structure. Inconsistent headings, boilerplate-heavy templates, or generated markup make structural boundaries meaningless. If your headings are decorative rather than semantic, splitting on them just adds noise.

Heterogeneous chunk sizes. Structure-first splitting produces wildly variable chunks. A one-line section and a ten-page section are both "one section." Oversized or mixed-topic units still need a secondary split, and tiny fragments may need merging.

Multi-page and multimodal content. Tables spanning pages, embedded figures, and procedural sequences need layout-aware or vision-assisted parsing, not text-only splitting. A text-only splitter will silently break a table that crosses a page boundary.

Versioning and permissions. When validity windows or access control matter, those constraints belong at candidate retrieval, not baked into chunk text. Filtering after retrieval wastes slots; filtering before retrieval is a metadata problem, not a chunking problem.

Overlap misuse. Stacking overlap on top of hierarchy prefixes duplicates context and wastes the attention budget. Pick one repair mechanism per failure type.

A Practical Chunking Pipeline

Assemble the pieces in sequence:

parse and clean
  -> split on native structure
  -> split only oversized or heterogeneous units
  -> attach title and hierarchy path
  -> index at multiple levels
  -> retrieve with hybrid search
  -> reconstruct parent or neighbor context

Start with the narrow version: structure-first splitting plus hierarchy metadata, evaluated against a fixed-window control. Add semantic chunking or rerankers only after that comparison tells you where the remaining failures live.

Two operational habits pay off. First, store canonical document state separately from the rebuildable search projection, so a chunking change does not require re-ingesting source documents. Second, version your chunk configuration alongside the chunk set, so a retrieval regression can be traced to a specific splitter change rather than discovered by archaeology.

The Decision Rule

If your source structure is trustworthy, split on it first and add hierarchy metadata before reaching for semantic chunking or larger overlap. Structure is authored information—headings and section boundaries were placed deliberately to group related material. Destroying them before testing them throws away a prior that a fixed splitter can only approximate.

Your next action is concrete: run the fixed-window control against a structure-first splitter on your own corpus. Measure evidence-span recall. Classify the failures before changing anything else. The failures will tell you which repair—metadata, hierarchy prefix, parent reconstruction, or a secondary split—actually earns its place.