引導式測驗 · 硬 等級

配對和 Elo 評估測驗

區分成對偏好與絕對分數、Elo 更新與 Bradley-Terry 估計以及排名與通用品質聲明。

問題 1 的 8

Which limitation can pairwise evaluation reduce compared with an absolute 1-to-5 score?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. Which limitation can pairwise evaluation reduce compared with an absolute 1-to-5 score?
    • It can reduce disagreements caused by different uses of a numeric scale
    • It removes all judge bias
    • 它保證了事實的準確性
    • It eliminates the need to sample prompts
  2. What real-world platform is cited as using pairwise comparisons at scale for model evaluation?
    • A stock trading exchange
    • Chatbot Arena
    • A weather forecasting service
    • A video streaming platform
  3. Where does the Elo rating system used for LLM comparisons originally come from?
    • Credit scoring
    • Search engine ranking
    • Chess rating
    • 天气预报模型
  4. What does the Bradley-Terry model estimate from pairwise outcomes?
    • A universal absolute quality score
    • The number of tokens in each response
    • A guarantee that every voter agrees
    • A latent strength parameter related to the probability of one item beating another
  5. Why is randomizing the order of outputs important when using an LLM-as-judge for pairwise comparisons?
    • It controls for position bias, where a judge might favor whichever output is shown first
    • It makes the comparison run faster
    • It reduces the API cost of the comparison
    • It is required by MCP protocol rules
  6. According to the deep dive, what should be treated cautiously about Elo ratings from very few comparisons?
    • That they are always higher than they should be
    • That they carry real uncertainty and shouldn't be treated as precise until enough comparisons accumulate
    • That they can only be computed for exactly two competitors
    • That they cannot be visualized in a leaderboard
  7. In an Elo update, what determines how much a rating changes after a single match?
    • The alphabetical order of the competitors' names
    • The total number of competitors in the leaderboard
    • The difference between the actual outcome and the expected outcome, scaled by a constant K
    • The length of the input prompt
  8. Why might a prompt engineer switch from 1-to-5 absolute scoring to pairwise comparison mid-project?
    • Because pairwise comparisons are always cheaper to run
    • Because absolute scoring is not supported by any tool
    • Because pairwise comparisons require no judge at all
    • Because absolute scores clustered near the top of the scale, giving little separation between prompt versions