LLM-as-a-judge
Also: LLM as a judge, model-graded evaluation, AI grader
Using one AI to grade another AI's work.
The fuller definition
LLM-as-a-judge uses a language model to score or compare the outputs of another system against a rubric, as a cheaper substitute for human review.
The reference set has ten relevant events. Six are found, four are missed, and two additional returned items are unsupported. This example measures extraction, not every aspect of a legal work product.
A concrete example
A second model checks whether each chronology row quotes a passage that actually supports it.
The distinction that matters
The grader has its own errors and biases. Calibrate it against human judgments on a sample before trusting its scores.
Related terms
Evaluation (eval)
Testing an AI tool on a set of tasks to see how well it actually does.
Reliability & securityBenchmark
A standard test used to compare models. A high score on it does not mean the model is good at your task.
Reliability & securityHallucination rate
How often a system makes things up, measured on a set of test tasks.
Where it is taught
Test an AI workflow on work you can assessSource and further reading
Anthropic · Evaluating AI agentsThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.