AI Glossary Term

What does “LLM-as-Judge” mean?

Definition

Using a language model to score or compare outputs from other models during evaluation.

Browse the full AI Glossary Explore all free AI guides

How LLM-as-Judge works

Example

To compare two summaries, a judge model is given the source text and a rubric on accuracy and clarity, then picks the better summary. A sample of its verdicts is checked by people.

Why it matters

It scales evaluation far beyond what people can review. Judges have biases, such as preferring longer answers, so they need calibration against human ratings.

Common questions

Can I trust it?

Reasonably for many tasks when checked against human judgments, but not blindly.

Known biases?

Preferences for longer answers, for the first option shown, and for its own style.

How to reduce bias?

Swap answer order, use clear rubrics, and compare against a human-labeled sample.

Large Language Model (LLM)A language model trained on massive text corpora to generate and analyze text.Unsupervised LearningLearning patterns from unlabeled data without explicit target outputs.Differential PrivacyA privacy technique that adds statistical noise so individual records cannot be reliably inferred from outputs.SentencePieceA language-agnostic tokenizer that learns subword units directly from raw text without pre-splitting on whitespace.MMLUA benchmark testing language models across 57 academic and professional subjects using multiple-choice questions.Loss FunctionA mathematical objective that quantifies prediction error during training.

Learn more in our free guides

ChatGPT & LLMs LLM Evaluations Multilingual LLMs Reversal Curse in LLMs

Test yourself

ChatGPT & LLMs Quiz LLM Evaluations Quiz Multilingual LLMs Quiz