Guide · AI quality
LLM evals for non-ML teams: measure quality, not vibes.
“It seems better” is how most teams decide whether an AI change is an improvement. It’s also how regressions reach production. Evaluations — evals — are simply automated tests for AI behaviour. You don’t need a data-science team to build useful ones; you need real examples, clear criteria and discipline.
By Nythrex EngineeringUpdated 3 min read
Why “try a few prompts” doesn’t work
LLM outputs vary, and improvements in one area often cause regressions in another. A change that fixes the three examples you tried may break twenty you didn’t. Without a fixed test set, every discussion about quality is opinion against opinion — and the loudest stakeholder wins.
Build your first eval set in a week
- 1
Collect real inputs
Pull 50–100 real questions, tickets or documents from the actual workflow. Include common cases, hard cases, and cases where the right answer is “I don’t know”.
- 2
Write what “good” means
For each case, write the reference answer or the key facts it must contain — together with the domain experts. This is the most valuable hour of the project.
- 3
Tag the cases
Label by type (product area, language, difficulty) so you can see where the system is weak, not just the average.
- 4
Automate the run
A script that runs every case through the system and stores outputs, scores, cost and latency per run.
- 5
Keep it alive
Every production failure becomes a new case. The set grows with real usage.
How to score outputs
| Method | Good for | Watch out for |
|---|---|---|
| Exact / rule-based checks | Classification labels, extracted fields, JSON format, required phrases | Brittle for free text |
| Reference comparison | Answers with known key facts | Paraphrases can be wrongly marked as wrong |
| LLM-as-judge | Free-text quality: correctness, faithfulness to sources, tone | Judges have biases — calibrate against human ratings |
| Human review | Calibration, ambiguous cases, final sign-off | Slow and expensive — use for samples |
| Retrieval metrics | RAG: did the right passage reach the context? | Needs labelled relevant passages |
Metrics that matter for most teams
- Correctness — is the answer right?
- Faithfulness — is every claim supported by the provided sources? (RAG)
- Refusal quality — does it decline when it should, and not when it shouldn’t?
- Format compliance — valid JSON, required fields, length limits.
- Safety — no leakage of personal data, no policy violations, resists prompt injection.
- Cost and latency per case — reported next to quality, every run.
Evals in CI: the habit that prevents regressions
- 01
Change
Prompt, model, retrieval or tool
- 02
Run eval set
Automatically, on every change
- 03
Compare
Against the last accepted run
- 04
Review diffs
Which cases got better or worse
- 05
Ship or fix
Thresholds decide
Treat the eval run like a test suite: if quality on a critical category drops below a threshold, the change doesn’t ship. When a model provider announces a new version, the same suite tells you in an hour whether switching is safe.
Frequently asked questions
Keep reading
Want a second opinion on your project?
Tell us what you’re building and where you’re stuck. We’ll reply within one business day with the most practical next step — even if that step isn’t us.