Guide-linked quiz · Medium Level
Statistical Power and Sample Size for Model Experiments Quiz
Calculate a sample-size approximation and identify how effect size, variance, power and experimental design influence required traffic.
Question 1 of 8
In the stated two-arm example, what sample size is approximately required per group?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- In the stated two-arm example, what sample size is approximately required per group?
- 392
- 196
- 784
- 80
- If the target MDE is halved with other assumptions fixed, how does approximate sample size change?
- It becomes half as large
- It becomes about four times larger
- It is unchanged
- It becomes one quarter as large
- What does 80% power mean under the specified alternative and assumptions?
- An 80% chance every user benefits
- An 80% chance the null is false
- An 80% chance the test rejects the null if that effect is present
- An 80% confidence interval coverage
- What does the MDE represent in planning?
- The maximum possible variance
- The effect guaranteed by the model
- The observed effect after launch
- The smallest effect chosen as important to detect
- Why does cluster randomization often require more observations?
- Each cluster has a separate p-value automatically
- Outcomes within a cluster are correlated and provide less independent information
- Clusters eliminate variance
- The treatment becomes continuous
- Why plan for multiple metrics or variants?
- Metrics cannot be correlated
- Power is always exactly zero for multiple metrics
- Many comparisons can increase false-positive risk
- More variants reduce all uncertainty
- Why is repeatedly checking fixed-horizon p-values and stopping at significance risky?
- It makes random assignment invalid
- It always lowers power
- It guarantees the effect is practical
- It can inflate false-positive rates unless a valid sequential method is used
- What should accompany a reported power calculation?
- Its assumptions about variance, test, allocation and effect size
- Only the chosen sample count
- The model's training accuracy
- A guarantee of causal generalization
Keep testing yourself
More quizzes picked for your level.