Evaluation & Quality
Prediction calibration
How closely an AI's stated chances match how often things actually happen.
Example
Of events given about an 80% chance, roughly eight in ten happen.
Why people use it
It helps people understand whether a confidence number deserves to be read as a chance.
What you'll hear
“When it says 80%, how often is it right?”
What this means for you
Ask whether the quoted chances have been checked against real results.
Can you control it?
No
No direct control. This describes a wider issue, concept or result rather than something you can simply switch on or off in a tool.
Common questions
- Can an AI choose the right group but give misleading chances?
- Yes. It might often choose correctly while sounding much more certain than the evidence warrants.
- Can good calibration change over time?
- Yes. Changes in the people or situations being assessed can make old probability estimates less trustworthy.
- Can two groups have different calibration?
- Yes. A system can give useful probability estimates for one group and misleading ones for another.