Calibrating LLM Judges Against Humans

Using an LLM as a judge is attractive because it scales far beyond what human review can cover, but a judge that has never been checked against human judgment is essentially untested. Calibration starts by collecting a set of examples that have already been rated by humans - ideally by more than one rater, so inter-human agreement provides context for interpreting judge-human agreement - and then running the LLM judge on those same examples using its intended rubric and prompt. The core comparison typically uses Cohen's kappa, a statistic that measures agreement between two raters while correcting for the agreement expected by pure chance, which matters because raw percent agreement can look deceptively high when one category is very common. A confusion matrix, which cross-tabulates human labels against judge labels for each category, then shows exactly where disagreement is concentrated - for example, a judge might agree with humans well on clear passes and clear failures but disagree heavily in a middle 'borderline' category. That pattern has several possible causes, including unclear rubric boundaries, judge-specific errors, or disagreement among human raters. Inspect examples before deciding whether to revise the rubric, add examples, adjust the judge, or adjudicate the reference labels. A common misconception is that a single round of calibration is sufficient forever; judges can drift as the underlying model, prompt, or the distribution of inputs being judged changes over time, so recalibration on fresh human-labeled samples on a regular cadence is standard practice. Another misconception is that perfect agreement is the bar to hit; human-human agreement is a useful reference for the task’s subjectivity, not a universal target or strict ceiling for judge performance.