Guide-linked quiz · Medium Level

Sequential Testing and the Peeking Problem Quiz

Explain why unplanned peeking changes error rates and compare fixed interim looks, alpha spending and always-valid inference.

Question 1 of 8

Why can stopping when an ordinary p-value first falls below 0.05 inflate false positives?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. Why can stopping when an ordinary p-value first falls below 0.05 inflate false positives?
    • Repeated looks create multiple chances to cross the threshold
    • The p-value stops being numeric
    • Randomization is removed automatically
    • The effect size becomes zero
  2. What does a group-sequential design specify in advance?
    • The exact future treatment effect
    • Interim analysis times and evidence boundaries
    • Every user's response
    • The final metric value
  3. What does an alpha-spending function allocate?
    • Model weights across layers
    • Sample labels across treatment groups
    • Type-I error across planned interim looks
    • Traffic costs across regions
  4. How do always-valid methods differ from unadjusted repeated p-values?
    • They remove the need for random assignment
    • They assume the test ends before data arrive
    • They guarantee every effect is real
    • They are designed for valid inference under data-dependent stopping
  5. Why define interim looks before seeing experiment results?
    • It makes the data independent
    • The schedule and boundaries determine the inference procedure
    • It guarantees early significance
    • It removes delayed labels
  6. What can early stopping do to an observed effect estimate?
    • Remove uncertainty
    • Guarantee the estimated effect is larger
    • Make an extreme early estimate less stable
    • Prevent multiple testing
  7. Which principle remains necessary after choosing a valid sequential stopping method?
    • Report the smallest ordinary p-value as if it were prespecified
    • Assume early estimates are precise because the test is sequential
    • Stop at any time without accounting for the chosen method
    • Prespecify randomization, metrics and handling of delayed outcomes
  8. What should a report include after a sequential experiment?
    • The stopping rule and timing and number of looks
    • Only the smallest p-value observed
    • The model's training loss
    • A statement that all future results will match

Keep testing yourself

More quizzes picked for your level.

View all quizzes