Guide-linked quiz · Medium Level

Statistical Significance in LLM Evals Quiz

Interpret evaluation score differences using matched test design, uncertainty intervals, effect size, and practical relevance for product decisions.

Question 1 of 8

What does a statistical significance test assess?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. What does a statistical significance test assess?
    • Whether a model is safe in every deployment
    • Whether users prefer a new interface
    • Whether an observed result would be surprising under a specified null hypothesis
    • Whether a score increase is large enough to matter
  2. Why use a paired comparison when both systems answer the same evaluation items?
    • It preserves item-level correspondence when estimating score differences
    • It guarantees the systems have identical scores
    • It makes every example independent
    • It removes all output randomness
  3. When may McNemar’s test be appropriate?
    • For any continuous score without assumptions
    • Only for latency measurements
    • For comparing unrelated samples
    • For paired binary outcomes on the same items
  4. What should be reported alongside a significance result?
    • Only which model won
    • Effect size, confidence interval, sample size, and test method
    • Only a p-value
    • A screenshot of the leaderboard
  5. Why can repeated model runs be useful?
    • They guarantee deterministic output
    • They replace a representative test set
    • They estimate run-to-run variability separately from variation across test examples
    • They always increase the mean score
  6. Does statistical significance prove a gain matters to users?
    • Yes, by definition
    • No; practical effect size and product goals must also be considered
    • Yes, if the p-value is small
    • No, because statistics cannot inform decisions
  7. What should analysts consider when testing many models and metrics?
    • Multiple-comparison risk or treating exploratory results as provisional
    • Ignoring all confidence intervals
    • Choosing only the largest score after testing
    • Changing the test set after seeing results
  8. How can teams reduce biased release decisions from a tiny score bump?
    • Ship every statistically higher score
    • Test only on examples used to tune the prompt
    • Ignore subgroup changes
    • Use a predeclared practical threshold and inspect representative held-out examples

Keep testing yourself

More quizzes picked for your level.

View all quizzes