Evaluation & Quality
Eval set
A collection of test examples used specifically to measure performance.
Example
A fixed collection of prompts and expected results is used for repeated evaluation.
Why it matters
It makes quality checks repeatable as a product changes.
You'll hear it
“Add that failed customer question to the checks we run next time.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Does a large eval set have to be representative?
- No. Many similar examples can still leave important situations uncovered.
- Can expected answers need updating?
- Yes. Changed facts, policies or task requirements can make earlier answers outdated.
- Should every failure be given equal importance?
- Not necessarily. Some errors have much greater practical consequences than others.