Evaluation & Quality
Evaluation
Eval
The process of measuring how well an AI system performs on defined tasks or criteria.
Example
A team tests whether a new assistant answers representative customer questions correctly.
Why it matters
It helps teams choose between tools and find weaknesses before more people depend on them.
You'll hear it
“The new assistant did well on easy questions but struggled with exceptions.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Can a high average hide problems for some users?
- Yes. Strong results on common cases can outweigh serious mistakes on a smaller group.
- Can the test examples become too familiar?
- Yes. Reusing them during development can make scores improve without showing equal improvement on unfamiliar work.
- Is one test run enough?
- Not always. Repeated runs can reveal variation that one successful attempt would miss.