How to Test RAG Retrieval Quality Before Rollout

An index that returns sensible results for the questions you tried is not a tested index. Testing retrieval means running a fixed set of questions with known answers, scoring every result the same way, and running the set again whenever anything changes. It is the only way to know whether a change helped.

This lesson tests the service desk knowledge base from the first lesson in this path, and compares three index settings with one test set. Every number below comes from those runs.

Step 1: Write a test set that includes hard questions

Start with the ten questions from the first lesson, written the way people actually ask. Then add questions whose answer sits near the end of a long article, where chunking is most likely to cut it off. We added six, one for each article longer than 500 characters.

Question	Expected article	Answer that must come back
I got a new phone and can’t approve sign-in requests	KB-1002	Require re-register MFA
Do MFA resets for finance staff need a second approver?	KB-1002	second service desk engineer
Can I reinstall the VPN client myself?	KB-1001	Do not reinstall
Who can give a delegate access to their manager’s calendar?	KB-1007	Editor or Reviewer
Do we get proof that an old laptop was destroyed?	KB-1005	certificate of destruction

The third column is the one most test sets leave out. Write the exact words the answer depends on, so you can check the passage as well as the article.

Step 2: Decide what counts as a pass

Score four things for every question, with the top three results:

• Right article first. The expected article is the top result.
• Right article in the top three.
• Answer in the top passage. The top result’s text contains the answer phrase.
• Answer in the top three passages.

The passage scores matter most, because a model is given passages, not articles. An article can rank first while the passage that ranked first holds none of the answer.

Step 3: Run it against the first index

You can run the set in Semantic Search, one question at a time with Top K set to 3, and record the results in a spreadsheet. We ran it as a script against the semantic search endpoint of the Data API, so every run is identical and takes seconds.

The first index uses the default settings, 500-character chunks with 50 characters of overlap, and holds 36 chunks. It put the right article first for 15 of 16 questions and in the top three for all 16. The answer was in the top passage for 8 of the 10 questions with an answer phrase, and in the top three passages for all 10.

Step 4: Read the failures, not just the totals

The totals look healthy. The failures behind them are where the lessons are.

For “Who can give a delegate access to their manager’s calendar?”, the right article came first, and its top passage was this 63-character fragment:

or or Reviewer permission set by the calendar owner, not by IT.

The chunker cuts at a fixed number of characters, not at sentence boundaries, and this cut fell inside the word “Editor”. Thanks to the 50-character overlap, “Editor or Reviewer” survived intact at the end of the first part, which ranked second. A model given only the top passage would answer “Reviewer” and miss “Editor”.

For “Do we get proof that an old laptop was destroyed?”, the wrong article came first: the lost or stolen device article at 0.601, just ahead of the replacement cycle article, which holds the answer, at 0.590.

One more check is worth doing on every run: compare the scores of right and wrong answers. The correct answer to “What priority is a whole site outage?” scored 0.531, lower than the wrong first result above. No fixed cut-off separates them, so a similarity threshold cannot be your quality gate.

Step 5: Change one setting and measure again

Change one thing at a time, and keep it only if the numbers improve. We built two more indexes over the same articles and ran the same set.

Index	Chunks	Right article first	Right article in top 3	Answer in top passage	Answer in top 3 passages
500 characters (default)	36	15/16	16/16	8/10	10/10
1,000 characters	30	15/16	16/16	9/10	10/10
1,000 characters with titles	30	14/16	16/16	8/10	10/10

At 1,000 characters with 100 of overlap, every article fits in one chunk, because the longest is 708 characters. The delegate question now returns the whole article, answer included.

The third index added each article’s title to its text, using the build API’s textColumns and textTemplate from the databases lesson. Titles were not being embedded, so it seemed an obvious improvement. It made retrieval worse: one fewer article ranked first, and the delegate question now returned the leaver process first. One likely reason is that the title names the article’s main problem, calendar sync, while the question is about the delegate rule at its end. We kept 1,000-character chunks and dropped the titles.

The destroyed-laptop question failed at every setting, which tells you it is not a chunking problem.

The answer was in the second result every time, so give the model the top three passages rather than only the first. Treat the failure as a content signal too: the replacement article answers the question, but not in the words people use to ask it. Rewording the article is the next change to measure.

Step 6: Keep the test set as a release gate

Keep the test set next to the index, and run it again whenever the content, the chunk settings, the embedding model or the source table changes. Agree on the gate before you need it. For example: every answer must appear in the top three passages, and no score may fall below the previous release. Record each run with its date and settings.

Add every question that fails in real use, with its answer phrase. After a few months the test set describes what your users actually ask, and it becomes the most valuable part of the pipeline.