Guide-linked quiz · Hard Level

Pairwise and Elo Evaluation Quiz

Distinguish pairwise preference from absolute scores, Elo updates from Bradley-Terry estimates, and rankings from universal quality claims.

Question 1 of 8

Which limitation can pairwise evaluation reduce compared with an absolute 1-to-5 score?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. Which limitation can pairwise evaluation reduce compared with an absolute 1-to-5 score?
    • It can reduce disagreements caused by different uses of a numeric scale
    • It removes all judge bias
    • It guarantees factual accuracy
    • It eliminates the need to sample prompts
  2. What real-world platform is cited as using pairwise comparisons at scale for model evaluation?
    • A stock trading exchange
    • Chatbot Arena
    • A weather forecasting service
    • A video streaming platform
  3. Where does the Elo rating system used for LLM comparisons originally come from?
    • Credit scoring
    • Search engine ranking
    • Chess rating
    • Weather forecasting models
  4. What does the Bradley-Terry model estimate from pairwise outcomes?
    • A universal absolute quality score
    • The number of tokens in each response
    • A guarantee that every voter agrees
    • A latent strength parameter related to the probability of one item beating another
  5. Why is randomizing the order of outputs important when using an LLM-as-judge for pairwise comparisons?
    • It controls for position bias, where a judge might favor whichever output is shown first
    • It makes the comparison run faster
    • It reduces the API cost of the comparison
    • It is required by MCP protocol rules
  6. According to the deep dive, what should be treated cautiously about Elo ratings from very few comparisons?
    • That they are always higher than they should be
    • That they carry real uncertainty and shouldn't be treated as precise until enough comparisons accumulate
    • That they can only be computed for exactly two competitors
    • That they cannot be visualized in a leaderboard
  7. In an Elo update, what determines how much a rating changes after a single match?
    • The alphabetical order of the competitors' names
    • The total number of competitors in the leaderboard
    • The difference between the actual outcome and the expected outcome, scaled by a constant K
    • The length of the input prompt
  8. Why might a prompt engineer switch from 1-to-5 absolute scoring to pairwise comparison mid-project?
    • Because pairwise comparisons are always cheaper to run
    • Because absolute scoring is not supported by any tool
    • Because pairwise comparisons require no judge at all
    • Because absolute scores clustered near the top of the scale, giving little separation between prompt versions

Keep testing yourself

More quizzes picked for your level.

View all quizzes