Guide-linked quiz · Medium Level

How Many Test Cases a Prompt Evaluation Needs Quiz

Plan prompt evaluation around the decision, detectable effect, outcome variability and representative sampling rather than a fixed count.

Question 1 of 8

Why is there no single sample size that is always enough for prompt testing?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. Why is there no single sample size that is always enough for prompt testing?
    • It depends on variability, effect size and the statistical design
    • Every prompt needs exactly 100 examples
    • It depends only on model parameter count
    • Confidence intervals eliminate case selection
  2. What does a confidence interval add to a measured pass rate?
    • A guarantee the next version will be better
    • An estimate of uncertainty around the measured rate
    • A list of training examples
    • A truth score for each answer
  3. Why compare versions on the same representative cases when possible?
    • To remove all output randomness
    • To make every case equally difficult
    • To enable paired comparison accounting for case-level differences
    • To increase the number of independent cases automatically
  4. What does repeating a prompt on one input measure most directly?
    • Population diversity
    • Coverage of the task distribution
    • Independent case count
    • Run-to-run variation for that input under those settings
  5. If a team oversamples rare safety failures, what should it avoid?
    • Reviewing those cases separately
    • Using them to discover failure modes
    • Treating the raw failure fraction as population prevalence without weighting
    • Labeling the sample as targeted
  6. Which inputs support a statistical sample-size plan?
    • Prompt word count only
    • Model training-set size
    • Number of evaluators
    • Target effect, outcome variability, confidence/significance and desired power
  7. What should a team define before comparing prompts?
    • The best early pilot outcome
    • Only which prompt looks polished
    • Sample size after seeing every result
    • Metrics, sampling frame, grading rules and decision threshold
  8. Given a small evaluation with a wide interval, how strong a conclusion is justified?
    • The prompt is proven to improve quality
    • Ignore the interval if the mean is higher
    • The result is uncertain and more representative evidence may be needed
    • Ship because no regression was observed

Keep testing yourself

More quizzes picked for your level.

View all quizzes