Guide · April 22, 2026
Evaluating an AI feature before you ship it
A prototype that looks good in a demo tells you almost nothing about production behaviour. Here is how to build an evaluation that does.
- AI
- Evaluation
- Quality
Solidslate AI · 9 min read
In our 2026 survey of engineering leaders, the teams with an automated evaluation set were far more likely to have AI in production. It is the clearest dividing line we found. This is how to build one.
Start with a labelled set, not a metric
Before choosing a score, collect fifty to a few hundred real examples of the task, with the input and a known good output. Draw them from actual usage or realistic scenarios, and deliberately include the hard cases: ambiguous inputs, adversarial ones, things at the edge of scope. This set is the asset. The metric comes after.
Score the way the product is judged
- For extraction or classification, exact or fuzzy match against the label
- For generation, a rubric applied by a stronger model, spot checked by a human until you trust it
- For retrieval, whether the right source was in the context, measured separately from the final answer
- Always track refusals and format failures as their own numbers
Run it offline on every change
The evaluation runs in the pipeline against a pinned version of the set. A prompt tweak, a model upgrade, a retrieval change: all of them produce a diff in the score before they merge. This is what turns AI work from guessing into engineering.
Freeze the set, version the changes
If the evaluation set changes at the same time as the system, you cannot tell whether the score moved because you improved the system or because you changed the exam. Treat the set like a schema.
Add online checks for what offline misses
- Log every input and output with enough context to replay it
- Sample production traffic daily back into the evaluation set
- Track cost per request and latency alongside quality, because a better answer that is too slow or too expensive is not better
- Put guardrails on input and output, and count how often they fire
Decide the bar before you look at the score
Agree what good enough to ship means in advance: the minimum score, the maximum acceptable rate of bad outputs, the cost ceiling. Deciding after you see the number is how features ship because everyone is tired of waiting.
A minimal setup
- Collect 100 real examples with known good outputs
- Write a scoring function and a rubric
- Wire it into CI against a frozen version of the set
- Set thresholds that block a merge
- Sample production back into the set weekly
Keep reading
Related pieces
Guide · March 10, 2026
A launch readiness checklist for production software
The checklist we run before putting anything in front of real users: reliability, security, observability, support and the rollback you hope not to need.
ReadField note · May 6, 2026
We deploy a walking skeleton before we write a feature
The first thing we put in production is a thin end to end slice that does almost nothing. Here is why that pays for itself in the first week.
ReadBuilding an AI feature?
Tell us what you're working on. We come back within two business days with a point of view and next steps.