Guide-linked quiz · Medium Level
Statistical Significance in LLM Evals Quiz
Interpret evaluation score differences using matched test design, uncertainty intervals, effect size, and practical relevance for product decisions.
Question 1 of 8
What does a statistical significance test assess?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- What does a statistical significance test assess?
- Whether a model is safe in every deployment
- Whether users prefer a new interface
- Whether an observed result would be surprising under a specified null hypothesis
- Whether a score increase is large enough to matter
- Why use a paired comparison when both systems answer the same evaluation items?
- It preserves item-level correspondence when estimating score differences
- It guarantees the systems have identical scores
- It makes every example independent
- It removes all output randomness
- When may McNemar’s test be appropriate?
- For any continuous score without assumptions
- Only for latency measurements
- For comparing unrelated samples
- For paired binary outcomes on the same items
- What should be reported alongside a significance result?
- Only which model won
- Effect size, confidence interval, sample size, and test method
- Only a p-value
- A screenshot of the leaderboard
- Why can repeated model runs be useful?
- They guarantee deterministic output
- They replace a representative test set
- They estimate run-to-run variability separately from variation across test examples
- They always increase the mean score
- Does statistical significance prove a gain matters to users?
- Yes, by definition
- No; practical effect size and product goals must also be considered
- Yes, if the p-value is small
- No, because statistics cannot inform decisions
- What should analysts consider when testing many models and metrics?
- Multiple-comparison risk or treating exploratory results as provisional
- Ignoring all confidence intervals
- Choosing only the largest score after testing
- Changing the test set after seeing results
- How can teams reduce biased release decisions from a tiny score bump?
- Ship every statistically higher score
- Test only on examples used to tune the prompt
- Ignore subgroup changes
- Use a predeclared practical threshold and inspect representative held-out examples
Keep testing yourself
More quizzes picked for your level.