Chunking is an information design problem

A paragraph that begins “this restriction does not apply” is perfectly understandable inside its document and almost useless on its own. Once retrieval separates it from the preceding rule, a model has to guess what “this” means. Chunking is therefore not simply a way to fit text into a context window. It decides which relationships remain visible.

Preserve the smallest useful explanation

Imagine a guide describing two subscription tiers. One section introduces the enterprise tier, and the next paragraph lists an exception to its retention policy. A fixed-size split might place the exception in a chunk with no mention of the tier. Searching for retention could now retrieve a technically accurate passage that invites the wrong interpretation.

A better starting point is the document's structure. Keep a heading with its explanation. Preserve list introductions and table headers where possible. Carry source identifiers and section paths as metadata. These choices do not guarantee better retrieval, but they give an experiment a clearer rationale than choosing a token count because it worked elsewhere.

Context has a cost

Adding neighbouring text can clarify a fragment. Adding too much can dilute the specific fact a query needs. The useful unit for an API reference may be one operation and its parameters; for a policy, it may be a rule together with its exceptions. There is no reason those units must have identical lengths.

Anthropic's contextual retrieval approach describes enriching chunks with document-specific context before indexing. The general lesson is that a passage's meaning can depend on information outside its boundary. If that context is generated, verify it: a mistaken summary attached to every retrieval result can propagate the same error repeatedly.

Treat tables and versions deliberately

A table row without its headers is an especially easy way to lose meaning. “30 / 90 / unlimited” tells a reader almost nothing without the labels and units. Convert or store the representation so those relationships survive retrieval, and inspect the actual extracted text rather than assuming the PDF parser preserved them.

Versioning matters for the same reason. Two chunks can describe the same setting and disagree because both were correct at different times. Store enough provenance to explain which document and revision support an answer. Decide explicitly whether historical questions are in scope instead of mixing every version into one undifferentiated index.

Let questions choose the experiment

Take a small set of representative questions and compare a few candidate strategies. Inspect missed evidence, irrelevant neighbours, and ambiguous excerpts. Record retrieval quality alongside index size and the amount of context ultimately sent to the model.

If larger chunks help only questions that need surrounding explanation, a targeted parent-section lookup may be worth testing. If exact identifiers keep disappearing, the problem might involve retrieval rather than boundaries. Chunking is one part of a system, not a universal remedy.

Good chunks are not merely short. They preserve enough of the original author's reasoning for a second reader—human or model—to understand what the passage actually says.

Further reading: Anthropic: Contextual Retrieval.