How to Test an LLM Feature: Evals, Hallucination Rates, and Release Gates
Build the eval set from real usage, not imagination
The most valuable testing asset you will own is a few hundred real inputs with agreed correct outputs. Pull them from actual user requests, support tickets and the awkward cases your team already argues about. Invented test cases cluster around what you expected, which is exactly where the system already works. Keep a held-out portion that never informs prompt tuning, so you have an honest measure rather than one you have gradually overfitted.
Grade with rules where you can and models where you must
Anything checkable deterministically should be — valid JSON, required fields present, values within range, citation present, the correct category chosen. Reserve model-based grading for the genuinely subjective parts like helpfulness and tone, and validate that grader against human ratings on a sample before trusting it. A model grader that disagrees with your reviewers 30% of the time is producing a number, not a measurement, and teams do sometimes optimise against it for weeks before checking.
Hallucination needs a definition specific to your product
'Made something up' is too vague to measure. In a document Q&A system, it might mean a claim not supported by any retrieved chunk. In a data assistant, a number not present in the query result. Define the failure precisely, then measure two things separately: how often it happens, and how often it happens without any hedge or citation that would let a user catch it. The second number is the one that predicts real-world harm, and it is usually much smaller and much more actionable.
Wire evals into CI
Every prompt change, retrieval change, model version bump and dependency update should trigger the eval suite, with results posted on the pull request and a threshold that blocks merge on regression. This turns prompt engineering from folklore into engineering. It also protects you from the specific pain of a provider updating a model underneath you: you find out from a failing build, not from a customer.
Monitor production, because evals go stale
Real inputs drift away from your test set within months. Sample production traffic continuously, have a human review a small percentage weekly, and feed newly discovered failures back into the eval set. Log user signals — thumbs down, corrections, escalations, abandoned sessions — and treat spikes as incidents. The eval set should grow every month; one that has not changed since launch is measuring last year's product.
Define the release gate up front
Decide before building what accuracy and unhedged-error rate are acceptable for this use case, and who signs off. A drafting assistant with a human editor can ship at a much lower bar than an automated decision system. Writing that down early prevents the launch-week conversation where nobody can say whether 91% is good, and it gives your team a target instead of an argument.
Have a project that needs this kind of thinking?
Let's talk through what you're building — free consultation, no commitment.
Get in Touch