Guide-linked quiz · Medium Level
How Many Test Cases a Prompt Evaluation Needs Quiz
Plan prompt evaluation around the decision, detectable effect, outcome variability and representative sampling rather than a fixed count.
Question 1 of 8
Why is there no single sample size that is always enough for prompt testing?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- Why is there no single sample size that is always enough for prompt testing?
- It depends on variability, effect size and the statistical design
- Every prompt needs exactly 100 examples
- It depends only on model parameter count
- Confidence intervals eliminate case selection
- What does a confidence interval add to a measured pass rate?
- A guarantee the next version will be better
- An estimate of uncertainty around the measured rate
- A list of training examples
- A truth score for each answer
- Why compare versions on the same representative cases when possible?
- To remove all output randomness
- To make every case equally difficult
- To enable paired comparison accounting for case-level differences
- To increase the number of independent cases automatically
- What does repeating a prompt on one input measure most directly?
- Population diversity
- Coverage of the task distribution
- Independent case count
- Run-to-run variation for that input under those settings
- If a team oversamples rare safety failures, what should it avoid?
- Reviewing those cases separately
- Using them to discover failure modes
- Treating the raw failure fraction as population prevalence without weighting
- Labeling the sample as targeted
- Which inputs support a statistical sample-size plan?
- Prompt word count only
- Model training-set size
- Number of evaluators
- Target effect, outcome variability, confidence/significance and desired power
- What should a team define before comparing prompts?
- The best early pilot outcome
- Only which prompt looks polished
- Sample size after seeing every result
- Metrics, sampling frame, grading rules and decision threshold
- Given a small evaluation with a wide interval, how strong a conclusion is justified?
- The prompt is proven to improve quality
- Ignore the interval if the mean is higher
- The result is uncertain and more representative evidence may be needed
- Ship because no regression was observed
Keep testing yourself
More quizzes picked for your level.