Evaluation & Quality
LLM-as-a-judge
Using a language model to score or compare outputs from another AI system.
Example
A model grades draft answers against a rubric before human spot-checking.
Why it matters
It helps review large collections of answers more quickly than checking every one manually.
You'll hear it
“The automated judge rewarded confident wording over a careful answer.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Can the judge make its own factual mistakes?
- Yes. It may misunderstand the evidence or accept a convincing unsupported claim.
- Can answer order affect its preference?
- It can. Some judges favor one position or presentation style.
- Does using a different AI guarantee independence?
- No. Systems can share blind spots, teaching material or preferences.