Chunking is treated as a preprocessing detail — a parameter set once during the pilot and never revisited. It is more accurately described as the point at which you decide what your retrieval system is capable of knowing. A fact that gets split across two chunks, or buried in a chunk that reads as unrelated, is a fact the system cannot reliably retrieve no matter how good the embedding model, the reranker or the generation step downstream.
This matters more in enterprise corpora than in the demo datasets chunking defaults were tuned on. Enterprise documents are long, structured, cross-referential and frequently not text at all. A 200-page policy manual, a master services agreement with 40 schedules, a maintenance procedure whose meaning lives in a table, a scanned form from 2014 — each defeats a naive character-count splitter in a different way.
The four boundary strategies, and when each is right
Fixed-size splitting cuts every document into N tokens with some overlap. It is cheap, predictable, and completely indifferent to meaning. It remains a reasonable fallback for genuinely unstructured text and a poor default for anything else, because most enterprise documents already tell you where the boundaries are.
Structural splitting cuts on the document’s own markers: headings, numbered clauses, slide boundaries, table rows, list items. When a document has real structure, this is almost always the right starting point — the author has already done the segmentation work, and discarding it in favour of a character count throws away information for free. A policy document chunked on clause boundaries retrieves clauses; the same document chunked at 512 tokens retrieves clause fragments.
Semantic splitting places boundaries where the similarity between adjacent sentences drops, following the shape of the argument rather than the formatting. It costs more at indexing time and earns its keep on narrative prose — reports, memos, correspondence — where the structure is rhetorical rather than typographic.
Late chunking embeds the full document with a long-context model first, then splits the resulting token sequence and pools each slice, so every chunk’s vector was computed with the whole document in view. It addresses the isolation problem directly and is constrained by model context and indexing cost.
The practical answer for most enterprise corpora is not one strategy but a router: structural where structure exists, semantic where it does not, page-level for scanned material, and special handling for tables. Which means the ingestion pipeline needs to know what kind of document it is holding — a classification step that is easy to skip and expensive to retrofit.
The isolation problem, and contextualisation
The recurring failure in enterprise RAG is not that chunks are the wrong size. It is that a chunk, read alone, is ambiguous about what it refers to.
“The notice period is 90 days” is a perfectly good chunk. It is also useless, because there are 140 contracts in the corpus and the chunk does not say which one it came from, or which party it binds, or which clause it sits under. Embedded on its own, it is semantically near every other notice-period sentence in the estate. Retrieved, it is a plausible-sounding answer to the wrong question.
The fix is contextualisation: before embedding, prepend a short statement of where the chunk sits — the document title, the heading path, the parties, the effective date, a one-line summary of the parent section. Anthropic’s published work on contextual retrieval reported that adding generated context to chunks before embedding, combined with contextual BM25, reduced failed retrievals by 49%, and by 67% when reranking was added on top.
Two practical notes. First, a large part of that benefit is available without a model call per chunk: deterministically prepending the document title and heading path costs nothing and resolves most of the “which document is this?” ambiguity. Start there, then measure whether generated context adds enough to justify the indexing cost on your corpus. Second, contextualisation interacts with access control — the prepended header must not leak information the retrieving user is not entitled to see, which means it is generated within the same metadata and permission model as the chunk itself.
Document types that need their own handling
Policies, procedures and regulations. Meaning is hierarchical and cross-referential: a clause qualified by a definition three sections earlier and an exception in an annex. Chunk on the numbering hierarchy, carry the full clause path in the chunk header, and preserve internal references so the retrieval layer can follow them. Also carry effective dates — a superseded policy retrieved as current is one of the more serious failure modes in a governed environment.
Contracts. Similar structure, harder stakes. The unit of retrieval is the clause, but the useful unit of answer is often the clause plus its definitions plus any amending schedule. Keeping the parties, agreement name and execution date in every chunk header is what prevents the cross-contract confusion described above.
Tables. The single most common silent failure. Split by character count, a table becomes chunks of numbers with no column headers, which embed poorly and generate confidently wrong answers. Keep small tables whole with their caption and headers; serialise larger ones row-wise with the header repeated in each row chunk; for large reference tables, index a description that points at the structured source and let the agent query it directly rather than retrieving it as prose. If the answer requires arithmetic across rows, retrieval is the wrong mechanism — that is a database query, not a document lookup.
Scanned documents and forms. The chunking decision is downstream of extraction, and extraction is where the corpus is won or lost. A layout model that reads a two-column page as one column produces text that is fluent and wrong. Use layout-aware extraction, keep the page as a fallback boundary, retain page numbers for citation so a reviewer can be pointed at the original image, and flag low-confidence extraction rather than indexing it silently.
Email and ticket threads. Chronological, quoted, and full of duplicated content — the same message body appearing in eight replies. Chunk per message rather than per thread, deduplicate quoted history, and keep sender, recipients and timestamp in the header. Otherwise the index fills with near-identical vectors that crowd out everything else.
Slide decks. One slide is one chunk, with the deck title and slide number, and speaker notes attached rather than indexed separately.
Measure on your own corpus, not on a benchmark
Every published comparison of chunking strategies is a measurement of someone else’s documents. The strategies that win in those studies win because of properties of the test corpus, and your corpus does not share them.
The minimum viable evaluation is a labelled question set — 50 to 100 real questions from the people who will use the system, each with the answering passage identified by a human. Then measure whether that passage appears in the retrieved set, per chunking configuration, per document type. The last dimension matters: an aggregate score hides a configuration that is excellent on policies and failing on anything containing a table. This is the evaluation harness described in measuring private RAG accuracy, and chunking is the variable it repays most.
One warning: changing the strategy means re-embedding the corpus. On a large estate that is a scheduled operation with a cost — an argument for evaluating on a representative sample early rather than discovering the problem after the full index is built and in use.
Why this runs inside the boundary
Chunking and contextualisation require reading every document in the corpus, in full, including the material that never gets retrieved. Where generated context is used, every chunk also passes through a model at indexing time. That is a far broader exposure than query-time retrieval — an ingestion run touches everything, not just what someone asked about.
Running the whole pipeline on infrastructure the organisation controls means the corpus, the intermediate extractions, the embeddings and the index all stay inside the existing boundary. It also makes re-indexing an operational decision rather than a procurement one: changing an embedding model or a chunking strategy does not involve sending the entire document estate to a third party again.
How VDF AI handles chunking
VDF AI’s ingestion runs entirely within the customer’s environment. Documents are processed with structure-aware extraction, chunk boundaries follow document structure where it exists, and chunks carry their source document, section path and access metadata so retrieval can be filtered by entitlement and citations can point back to the original. Embedding and reranking models run locally alongside the platform, so re-indexing after a strategy or model change is an internal operation. Chunking configuration is set per source rather than globally, which is what allows a policy library, a contract repository and a scanned archive to be handled correctly in the same knowledge base.