LLM-as-a-Judge

LLM-as-a-judge uses a language model to evaluate model outputs, including outputs from the same model. Open-ended tasks may admit several acceptable answers, making exact reference matching insufficient. A judge can apply criteria to many outputs at lower cost than some forms of expert review, but speed does not establish correctness or deterministic scoring. Google’s evaluation overview presents model-based evaluation as a scalable option that still needs validation for the intended task.

Forms

A pointwise judge rates one output; a pairwise judge compares two outputs for the same input. Either can use an assessment rubric with binary checks or ordered ratings. Explicit criteria clarify the judgment, but applying them still requires interpretation. Pairwise comparisons can be useful for candidate selection; their reliability depends on the task, instructions, presentation, and evaluator rather than on the format alone.

Known biases

Zheng and colleagues documented position and verbosity effects and limitations in grading reasoning. Position preference varied by model and prompt. They observed some models favoring their own outputs, but the paper states that limited data and small differences prevented a conclusion about self-enhancement bias. Its reported human agreement on the studied benchmarks is useful evidence for those settings, not a guarantee for another task or an entire model family.

To investigate position effects, present the same outputs in both orders and map each verdict back to the original output identities. Decide in advance how to aggregate results, including disagreement, abstention, and genuine ties. Google’s response-flipping configuration swaps baseline and candidate in part of the judge calls; it is separate from a policy that converts inconsistent verdicts into ties. Zheng’s S1 analysis uses that tie convention, but it is not a universal product contract. Report inconsistent judgments separately. Repeated sampling can reduce random variation but may reproduce systematic errors. Test verbosity and model-origin sensitivity on the task; a different-family judge is a candidate mitigation to validate, not proof of independence.

Validate the judge before using it

Validate the judge on reference-rated outputs not used to choose or tune it. Define whether the reference is an individual, a consensus, or an adjudicated rating, and inspect reference errors as well as judge errors. Report confusion patterns, missed critical failures, segment performance, and uncertainty. A grader that accepts every output can reach 95% agreement on 95 acceptable and five unsafe outputs while missing all unsafe cases. inter-annotator agreement supplies context, not a ceiling: two raters who make different errors can agree less with each other than a correct judge agrees with either. Investigate disagreements using source evidence rather than presuming either side is right.

Validation applies to the tasks and output populations studied; a new language or criterion needs suitable evidence. Record judge and rubric versions, prompt, settings, and data. Fixed identifiers do not remove randomness or all serving changes. If the evaluator changes, rescore stored baseline and candidate outputs under a common evaluator, and where possible use both old and new evaluators on the same outputs to separate grading changes from answer changes. Factual criteria need accessible reference evidence, with the limitations described in groundedness. The Judge Preferred Longer Answers connects these checks to human review and online measurement.

References:

• Google Cloud, Evaluate a judge model (Creative Commons Attribution 4.0).
• Google Cloud, Configure a judge model (Creative Commons Attribution 4.0).
• Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track.
• Google Cloud, Gen AI evaluation service overview.