Designing Production-Grade RAG Pipelines

A useful retrieval-augmented generation system is a chain of evidence handling decisions. It turns source material into searchable representations, selects material for a question, and asks a model to explain what that material supports. Weakness in any stage can appear to the user as the same problem: an answer they cannot trust.

Begin with the source of truth

Consider a documentation assistant for a software product. Before selecting a vector database, establish which documents are authoritative, how revisions arrive, and what should happen when a page is removed. An excellent index of obsolete documentation is still an obsolete system.

Give each source a stable identity and retain enough provenance to locate the original passage. Keep access information attached through ingestion and retrieval. A copied chunk should not lose the permission rules that governed its source document.

Keep ingestion inspectable

Extraction is an engineering boundary of its own. Headings can disappear, tables can flatten into confusing text, and scanned pages may need recognition before indexing. Inspect representative extracted output rather than judging the pipeline only by whether it produced embeddings.

Then choose chunking strategies that preserve useful relationships. A small passage may need its heading or surrounding explanation to be understandable. Store the source version, section path, and extraction status so a poor answer can be traced back to its input rather than treated as an unexplained model failure.

Retrieve for the question you actually have

Semantic retrieval can help connect differently phrased questions and passages. Exact names, identifiers, and error codes may also benefit from lexical matching. Combining approaches is worth evaluating, but it is not an automatic improvement for every collection.

The right experiment depends on observed misses. If relevant passages are found but ranked too low, reranking may help. If the required document never entered the index, no ranking strategy can recover it. Anthropic's contextual retrieval article explores one way to retain more document context; test such changes against your own questions rather than assuming published results transfer unchanged.

Give generation a bounded responsibility

The generation step should explain the supplied evidence, cite it where appropriate, and acknowledge when it cannot support an answer. Validate that citations point to real retrieved sources and inspect whether those sources actually support the claims.

Keep an evaluation set that separates retrieval failures from answer failures. Track latency and cost alongside quality, and include unsupported questions rather than only easy lookups. The architecture becomes easier to improve when each stage has a responsibility you can observe.

A production-minded RAG system is not defined by how many components it contains. It is defined by whether its evidence can be maintained, its mistakes investigated, and its answers checked.

Further reading: Anthropic: Contextual Retrieval.