Evaluating AI Systems Before and After Release
A fictional claims assistant earns a higher judge score but misses required facts. Evaluation needs independent evidence, validated ratings, and outcome checks.
-
-
Evaluating Agent Actions and Workflow Outcomes
A final reply can be right while the world is wrong. Score agent runs by the effect ledger, constrain the trajectory, test faults and attacks, audit reviewers, and gate releases.
-
Evaluating Retrieval and Answers Separately
One accuracy number cannot say whether a RAG failure came from retrieval, context selection, or generation. Score each stage, audit the judges, and diagnose by stage.
-
Reproducible Test Sets for AI Evaluation
A 27-versus-25 score on 30 pasted questions proves little. Build a test set from a failure taxonomy, sample it by stratum, score with checks and validated judges, and compare releases pairwise.