Calibration
Also: calibrated
Whether a system's stated confidence matches how often it is actually right. 'Ninety percent sure' should be right nine times in ten.
The fuller definition
Calibration describes how well predicted probabilities correspond to observed frequencies for a defined task and population.
Invented numbers. Each bar is the confidence the system stated; the darker fill is how often it was actually right. A well-calibrated system fills each bar to its stated level. This one is overconfident at 90% and underconfident at 50%, which is why a self-reported percentage is not a probability until it has been measured.
A concrete example
Among many cases assigned 80% probability, roughly 80% should succeed if those probabilities are well calibrated.
The distinction that matters
A model saying “80% confident” is not automatically a calibrated probability.
Related terms
Evaluation (eval)
Testing an AI tool on a set of tasks to see how well it actually does.
Reliability & securityPrecision
Of the things the system returned, how many were right.
Reliability & securityHallucination
When the model states something that sounds right but is made up, including cases, quotes, and facts that do not exist.
Where it is taught
Test an AI workflow on work you can assessSource and further reading
Anthropic · Evaluating AI agentsThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.