Guide-yakabatana mibvunzo · Pakati Nhanho

How Many Test Cases a Prompt Evaluation Needs Quiz

Plan prompt evaluation around the decision, detectable effect, outcome variability and representative sampling rather than a fixed count.

Nzira dzekutungamira dzinoenderanaSample Size For Prompt Evaluations
Mubvunzo 1 ye 8

Why is there no single sample size that is always enough for prompt testing?

Tarisa mubvunzo wese8 mibvunzo

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

  1. Why is there no single sample size that is always enough for prompt testing?
    • It depends on variability, effect size and the statistical design
    • Every prompt needs exactly 100 examples
    • It depends only on model parameter count
    • Confidence intervals eliminate case selection
  2. What does a confidence interval add to a measured pass rate?
    • A guarantee the next version will be better
    • An estimate of uncertainty around the measured rate
    • A list of training examples
    • A truth score for each answer
  3. Why compare versions on the same representative cases when possible?
    • To remove all output randomness
    • To make every case equally difficult
    • To enable paired comparison accounting for case-level differences
    • To increase the number of independent cases automatically
  4. What does repeating a prompt on one input measure most directly?
    • Population diversity
    • Coverage of the task distribution
    • Independent case count
    • Run-to-run variation for that input under those settings
  5. If a team oversamples rare safety failures, what should it avoid?
    • Reviewing those cases separately
    • Using them to discover failure modes
    • Treating the raw failure fraction as population prevalence without weighting
    • Labeling the sample as targeted
  6. Which inputs support a statistical sample-size plan?
    • Prompt word count only
    • Model training-set size
    • Number of evaluators
    • Target effect, outcome variability, confidence/significance and desired power
  7. What should a team define before comparing prompts?
    • The best early pilot outcome
    • Only which prompt looks polished
    • Sample size after seeing every result
    • Metrics, sampling frame, grading rules and decision threshold
  8. Given a small evaluation with a wide interval, how strong a conclusion is justified?
    • The prompt is proven to improve quality
    • Ignore the interval if the mean is higher
    • The result is uncertain and more representative evidence may be needed
    • Ship because no regression was observed