Before another prompt, build an evaluation

A retrieval-augmented answer can sound convincing while being wrong for two entirely different reasons. The system may never have found the right evidence, or it may have found it and misunderstood it. Editing the prompt before distinguishing those failures is like refocusing a camera without checking whether the subject is in the frame.

Start with questions that can fail

Consider an internal documentation assistant. A useful evaluation might include a straightforward setup question, an ambiguous product name, a question about a retired feature, and a question the documentation cannot answer. These cases exercise different responsibilities. The final one tests whether the assistant can stop rather than improvise.

For each case, record the question, the evidence a reviewer would need, and the conditions for a satisfactory answer. Do not require one exact sentence when several explanations would be acceptable. For a timeout setting, the condition might be “names the correct configuration key, describes its unit, and cites the current page.” That is more useful than asking whether an answer sounds helpful.

Examine retrieval separately

Run the question through retrieval and look at the returned passages before generation. Is the required evidence present? Is it current? Is the assistant allowed to use it? A passage can be semantically relevant yet belong to a different product version or another customer's workspace.

Suppose the correct document appears eighth but only five passages reach the model. That observation suggests experimenting with retrieval, filtering, or reranking. It does not establish that a longer prompt will help. Conversely, if the right paragraph is already first, changing the embedding model may leave the actual failure untouched.

Evaluate the answer against evidence

Now examine whether the answer follows from the supplied material. Treat factual correctness, citation support, and completeness as separate judgments. An answer can cite a genuine page that does not support its central claim. It can also be correct but omit a necessary exception.

Automated graders can accelerate this review, but they need calibration against human judgments. Keep examples of disputed ratings and refine the rubric. Anthropic's evaluation guide discusses the distinction between the task, the recorded trajectory, and the final outcome; that separation is helpful when an apparently good final answer hides a poor process.

Change one thing and keep the difficult cases

Use the same evaluation set to compare a baseline and a proposed change. Report failures by type rather than hiding them in one average. A gain on easy lookups should not conceal a regression on unsupported questions.

Keep a separate held-out set for checking whether improvements generalise. Otherwise, repeated tuning can turn the evaluation into another training target. When a new failure appears, investigate it, create a representative regression case, and protect any private information before saving it.

The point is not to produce a perfect score. It is to replace “this answer feels better” with a specific explanation of what improved, what did not, and what the system still cannot reliably do.

Further reading: Anthropic: Demystifying evals for AI agents.