Guide-linked quiz · Medium Level

Statistical Power and Sample Size for Model Experiments Quiz

Calculate a sample-size approximation and identify how effect size, variance, power and experimental design influence required traffic.

Question 1 of 8

In the stated two-arm example, what sample size is approximately required per group?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. In the stated two-arm example, what sample size is approximately required per group?
    • 392
    • 196
    • 784
    • 80
  2. If the target MDE is halved with other assumptions fixed, how does approximate sample size change?
    • It becomes half as large
    • It becomes about four times larger
    • It is unchanged
    • It becomes one quarter as large
  3. What does 80% power mean under the specified alternative and assumptions?
    • An 80% chance every user benefits
    • An 80% chance the null is false
    • An 80% chance the test rejects the null if that effect is present
    • An 80% confidence interval coverage
  4. What does the MDE represent in planning?
    • The maximum possible variance
    • The effect guaranteed by the model
    • The observed effect after launch
    • The smallest effect chosen as important to detect
  5. Why does cluster randomization often require more observations?
    • Each cluster has a separate p-value automatically
    • Outcomes within a cluster are correlated and provide less independent information
    • Clusters eliminate variance
    • The treatment becomes continuous
  6. Why plan for multiple metrics or variants?
    • Metrics cannot be correlated
    • Power is always exactly zero for multiple metrics
    • Many comparisons can increase false-positive risk
    • More variants reduce all uncertainty
  7. Why is repeatedly checking fixed-horizon p-values and stopping at significance risky?
    • It makes random assignment invalid
    • It always lowers power
    • It guarantees the effect is practical
    • It can inflate false-positive rates unless a valid sequential method is used
  8. What should accompany a reported power calculation?
    • Its assumptions about variance, test, allocation and effect size
    • Only the chosen sample count
    • The model's training accuracy
    • A guarantee of causal generalization

Keep testing yourself

More quizzes picked for your level.

View all quizzes