Quiz collegato alla guida · Medio Livello
How Many Test Cases a Prompt Evaluation Needs Quiz
Plan prompt evaluation around the decision, detectable effect, outcome variability and representative sampling rather than a fixed count.
Domanda 1 di 8
Perché non esiste una dimensione del campione unica che sia sempre sufficiente per eseguire test tempestivi?
Anteprima di ogni domanda8 domande
Le risposte e le spiegazioni vengono visualizzate mentre rispondi al quiz.
- Perché non esiste una dimensione del campione unica che sia sempre sufficiente per eseguire test tempestivi?
- It depends on variability, effect size and the statistical design
- Every prompt needs exactly 100 examples
- It depends only on model parameter count
- Confidence intervals eliminate case selection
- Cosa aggiunge un intervallo di confidenza al tasso di superamento misurato?
- A guarantee the next version will be better
- An estimate of uncertainty around the measured rate
- A list of training examples
- A truth score for each answer
- Why compare versions on the same representative cases when possible?
- To remove all output randomness
- To make every case equally difficult
- To enable paired comparison accounting for case-level differences
- To increase the number of independent cases automatically
- Cosa misura più direttamente la ripetizione di un prompt su un input?
- Population diversity
- Coverage of the task distribution
- Independent case count
- Run-to-run variation for that input under those settings
- Se un team esegue un campionamento eccessivo di rari guasti alla sicurezza, cosa dovrebbe evitare?
- Reviewing those cases separately
- Using them to discover failure modes
- Treating the raw failure fraction as population prevalence without weighting
- Labeling the sample as targeted
- Quali input supportano un piano statistico sulla dimensione del campione?
- Prompt word count only
- Model training-set size
- Number of evaluators
- Target effect, outcome variability, confidence/significance and desired power
- What should a team define before comparing prompts?
- The best early pilot outcome
- Only which prompt looks polished
- Sample size after seeing every result
- Metrics, sampling frame, grading rules and decision threshold
- Given a small evaluation with a wide interval, how strong a conclusion is justified?
- The prompt is proven to improve quality
- Ignore the interval if the mean is higher
- The result is uncertain and more representative evidence may be needed
- Ship because no regression was observed