Kvíz spojený s průvodcem · Střední úroveň

How Many Test Cases a Prompt Evaluation Needs Quiz

Plan prompt evaluation around the decision, detectable effect, outcome variability and representative sampling rather than a fixed count.

Související vodicí cestySample Size For Prompt Evaluations
Otázka 1 z 8

Why is there no single sample size that is always enough for prompt testing?

Náhled každé otázky8 otázky

Odpovědi a vysvětlení se zobrazují, jak budete v kvízu.

  1. Why is there no single sample size that is always enough for prompt testing?
    • It depends on variability, effect size and the statistical design
    • Every prompt needs exactly 100 examples
    • It depends only on model parameter count
    • Confidence intervals eliminate case selection
  2. What does a confidence interval add to a measured pass rate?
    • A guarantee the next version will be better
    • An estimate of uncertainty around the measured rate
    • A list of training examples
    • A truth score for each answer
  3. Why compare versions on the same representative cases when possible?
    • To remove all output randomness
    • To make every case equally difficult
    • To enable paired comparison accounting for case-level differences
    • To increase the number of independent cases automatically
  4. What does repeating a prompt on one input measure most directly?
    • Population diversity
    • Coverage of the task distribution
    • Independent case count
    • Run-to-run variation for that input under those settings
  5. If a team oversamples rare safety failures, what should it avoid?
    • Reviewing those cases separately
    • Using them to discover failure modes
    • Treating the raw failure fraction as population prevalence without weighting
    • Labeling the sample as targeted
  6. Which inputs support a statistical sample-size plan?
    • Prompt word count only
    • Model training-set size
    • Number of evaluators
    • Target effect, outcome variability, confidence/significance and desired power
  7. What should a team define before comparing prompts?
    • The best early pilot outcome
    • Only which prompt looks polished
    • Sample size after seeing every result
    • Metrics, sampling frame, grading rules and decision threshold
  8. Given a small evaluation with a wide interval, how strong a conclusion is justified?
    • The prompt is proven to improve quality
    • Ignore the interval if the mean is higher
    • The result is uncertain and more representative evidence may be needed
    • Ship because no regression was observed