LLM-as-a-Judge
A model can grade thousands of answers overnight. It can also favor longer answers or the one shown first. Check the judge before trusting its scores.
-
-
Evaluating AI Systems Before and After Release
A fictional claims assistant earns a higher judge score but misses required facts. Evaluation needs independent evidence, validated ratings, and outcome checks.
-
Online Evaluation and Controlled Rollout
An offline gate passes, but customers rephrase, outcomes arrive late, and thumbs-down rates mislead. Measure task outcomes, size each rollout stage, and write rollback criteria before exposure.
-
Prompt and Model Promotion with Evaluation Gates
A prompt or model change should reach users only with evidence. This guide builds a release gate service: versioned evaluation sets, non-inferiority thresholds, paired bootstrap intervals, slice checks, judge calibration, approvals, canary stages, and rollback, with runnable simulations.