LLM evaluation
How to Test a Policy Answering System with Retrieval
Test current policy retrieval, answer support, citations, and abstention with a worked failure trace.
A policy assistant should answer from the current policy and say when the answer is absent. Test whether the right passage was found, whether the answer used it, and whether the citation proves the claim. A system can cite a passage faithfully and still be wrong if that passage is outdated. This is a method, not a BuildAI RAG test.
Set the policy rule
The policy owner chooses the rule in this example: no known wrong or unsupported policy answer may be shown without review during the pilot. At least 95% of answerable questions should contain the right current citation, and at least 95% of unanswerable questions should receive an honest “I don't know” or handoff. The limits reflect this team's support workflow; high-stakes policies may require every answer to be reviewed.
Make freshness visible in the test set
Label real questions with the exact current policy passage and version. Add questions whose answer changed between old and new versions, and questions with no answer in the documents. Include PDF and web sources if both are used. Keep these questions out of prompt and retrieval tuning. Record parsing failures, index date, top-k passages, answer, and cited passage. Ragas's faithfulness metric checks consistency with retrieved context; it does not by itself decide whether that context is current policy.
Worked example: trace a wrong answer
Suppose a policy changed the refund window from 30 days to 14 days. The user asks, “Can I return this after 20 days?” The retriever returns an old 30-day PDF at rank 1 and the current 14-day page at rank 5. The assistant answers “yes” and cites the old PDF. The answer is supported by its citation but wrong under current policy. The fix starts with source versioning and retrieval, not just a stronger answer prompt.
| Test group | Count | Correct current citation or abstention | Example pilot rule |
|---|---|---|---|
| Current-policy questions | 80 | 77 | At least 76/80 |
| Changed-policy questions | 20 | 17 | Review every stale answer |
| No-answer questions | 20 | 19 abstentions | At least 19/20 |
Do not launch unreviewed because the changed-policy slice has three wrong answers. Remove or mark old pages, rebuild the index, and retest on a fresh changed-policy set. Twenty questions in a slice cannot establish that stale answers are rare. Keep human review until a much larger sample across policy changes supports the decision. If failures are from missing text in a scanned PDF, fix parsing first.
Report retrieval hit at top-k, answer correctness, citation match, abstentions, stale-source errors, latency, and cost by question group. Save a trace for every wrong answer so the owner can see which stage failed.
Download the results sheet. Fill one row for each test item; keep the item-level errors, not just the average.