Building a Golden Set to Catch RAG Regressions Before Users Do

A team writing test queries from scratch tends to write queries that are easier than what real users ask: cleanly phrased, on-topic, unambiguous. Pull your golden set from actual query logs instead, including the awkwardly phrased ones and the ones that returned poor results, since those are exactly the cases most likely to regress silently in the future. Aim for a set that reflects the real distribution of query types your system sees, not just the ones that are easy to write expected answers for.

Include a handful of queries the system should decline to answer, too, ones that fall outside your corpus entirely. A pipeline that confidently generates a plausible-sounding answer for a question it has no real data on is a failure mode a golden set built only from answerable queries will never catch.

For retrieval, correct usually means a specific set of chunk IDs should appear in the top results. For generation, correct is softer, usually a list of facts that must appear in the answer, and facts that must not. Write these definitions down manually for each query in the set before building any scoring automation, since agreeing on what correct means is a harder problem than building the tooling to check it, and skipping this step produces a set that scores things nobody actually agreed matter.

A high generation score can mask a retrieval problem if the underlying model already knows the answer from its training data and doesn't actually need the retrieved context to get it right. Score retrieval on its own with recall or mean reciprocal rank against your known-good chunk IDs, and score generation separately with a groundedness check against what was actually retrieved. Only looking at end-to-end answer quality hides exactly the failure mode a golden set exists to catch, and it's the split that tends to reveal which half of the pipeline actually needs the next round of work.

For example, suppose a change to chunk size lifts end-to-end answer quality slightly, so the team ships it. Scored separately, retrieval recall fell on queries that mention specific product codes, but the model answered them from its training data anyway. The generation score hid the problem until the pipeline met a question the model didn't already know. A split score would have flagged the drop at review time, when reverting was cheap, and the affected queries would then be added to the set as permanent regression cases.

An absolute quality threshold stops being meaningful as your corpus grows and query patterns shift, since a harder, more diverse corpus naturally scores lower even with a well-tuned pipeline. Compare each change against your previous version's score on the same golden set instead, and treat any meaningful drop as a blocker. This also makes the gate resilient to your corpus changing over time in ways that have nothing to do with a specific code or configuration change.

Every production complaint that turns out to be a genuine retrieval or generation miss is a candidate for a new golden set entry. Building this habit into your team's process, add the failing query, its correct answer, and a regression test in the same pass as the fix, keeps the set relevant instead of frozen at whatever the pipeline looked like on day one, and it's usually the cheapest source of new test cases you have.

A minimal golden set workflow looks like this:

1. Pull real queries from your logs, including awkward ones, poorly answered ones, and a few the corpus can't answer at all.
2. Write down what correct means for each query: expected chunk IDs for retrieval, and facts that must and must not appear for generation.
3. Score retrieval and generation separately, using recall or mean reciprocal rank for one and a groundedness check for the other.
4. Compare every change against the previous version's score on the same set, and treat a meaningful drop as a blocker.
5. Add each confirmed production miss to the set in the same pass as its fix.

Two engineers will occasionally disagree about whether a given answer is acceptable, especially for open-ended queries where more than one phrasing could be considered right. Without a named owner for the golden set, these disagreements either stall the set's growth or get resolved inconsistently case by case. Give one person, or a small rotating group, final say on borderline calls, and record the reasoning briefly next to the entry so the next disagreement over similar territory has precedent to work from, instead of relitigating the same judgment call every time it resurfaces.