AI features are probabilistic, so teams often skip testing and rely on manual checks. That works until a prompt change quietly breaks the cases nobody remembered. A small eval stack is not a platform project. It is a folder of cases and a runner you trust.
Start with a golden set
Write down 20 to 50 real tasks with the expected outcome. Include normal cases, edge cases and cases where the correct answer is 'I do not know'. Store them in the repository next to the prompt, in YAML or JSON, so a change and its test are reviewed together.
- id: refund-policy-01
input: "Can I get a refund after 40 days?"
must_contain: ["30 days", "not eligible"]
must_not_contain: ["yes, anytime"]
tags: [policy, edge]Use three kinds of assertions
Deterministic checks are cheap and stable: required phrases, valid JSON, a URL from an allowlist, a tool call with the right arguments. Rubric checks score qualities like grounding, completeness and tone. Human review samples the rest. Do not ask an LLM judge to decide things a regex can decide.
| Check | Use for | Cost |
|---|---|---|
| Exact / contains | Policy, format, required fields | Very low |
| Schema validation | JSON, tool arguments | Very low |
| Rubric + LLM judge | Tone, completeness, grounding | Medium |
| Human spot check | Ambiguous, high-stakes cases | High |
Calibrate the judge
An LLM judge has biases: it prefers longer answers, its own style, and the first option in a pairwise comparison. Label 30 examples yourself, compare the judge's scores to yours, and adjust the rubric until it agrees most of the time. Keep the judge prompt versioned like any other prompt.
Make it a CI gate
Run the eval set on every prompt, model or retrieval change. Fail the build if the pass rate drops beyond a threshold, or if a must-pass case fails. Also assert cost and latency budgets: an eval that ignores a 40% cost increase is only half a test.
pytest evals/ -q
# fail if pass rate < 0.9 or p95 latency > 3.0s
python evals/report.py --min-pass 0.90 --max-p95 3.0A worked run
A prompt change looked harmless: it added one sentence asking the model to be concise. The overall pass rate moved from 91% to 88%, which is inside the noise for a 30-case set. The interesting part was which cases moved.
| Case | Before | After | Diagnosis |
|---|---|---|---|
| Refund after 40 days | Pass | Fail | Dropped the 'not eligible' phrase |
| Parental leave | Pass | Pass | No change |
| No answer in corpus | Pass | Pass | Still refused correctly |
| Multi-doc policy | Pass | Fail | Stopped citing the second document |
The concise instruction made the model drop required qualifiers. That is a real regression even though the average barely moved. A must-pass list would have caught it immediately; a single aggregate score would not. Keep both: an overall pass rate for direction, and a small set of cases that must never fail.
Sample production
A golden set only covers what you thought of. Sample real runs weekly, especially failures and low-confidence answers, and promote the interesting ones into the set. This is how the eval suite stays connected to actual usage.
What to remember
- Keep 20–50 real cases in the repo next to the prompt.
- Use deterministic checks first; reserve LLM judges for qualities a regex cannot score.
- Calibrate the judge against human labels and version its prompt.
- Gate prompt/model changes in CI on pass rate, cost and latency.
- Promote production failures into the golden set every week.