AI Glossary Term
What does “Eval Harness” mean?
Definition
A repeatable evaluation framework that runs prompts, datasets, and scoring logic across model versions.
How Eval Harness works
Example
Before shipping a new prompt, a team runs it against 500 saved test questions and compares scores with the previous version. If scores drop on a category, they investigate before release.
Why it matters
Repeatable tests catch regressions that spot-checking misses. They make model and prompt changes safer to ship.
Common questions
What goes into a harness?
Datasets, prompt templates, model calls, scoring logic, and a report.
Does it replace human review?
No. It handles routine checks at scale; people still review samples and edge cases.
Are there open tools?
Yes. Several open-source evaluation frameworks exist, and many teams also build their own.