Quiz collegato alla guida · Medio Livello
Statistical Significance in LLM Evals Quiz
Interpret evaluation score differences using matched test design, uncertainty intervals, effect size, and practical relevance for product decisions.
Domanda 1 di 8
Cosa valuta un test di significatività statistica?
Anteprima di ogni domanda8 domande
Le risposte e le spiegazioni vengono visualizzate mentre rispondi al quiz.
- Cosa valuta un test di significatività statistica?
- Whether a model is safe in every deployment
- Whether users prefer a new interface
- Whether an observed result would be surprising under a specified null hypothesis
- Whether a score increase is large enough to matter
- Perché utilizzare un confronto appaiato quando entrambi i sistemi rispondono agli stessi elementi di valutazione?
- It preserves item-level correspondence when estimating score differences
- It guarantees the systems have identical scores
- It makes every example independent
- It removes all output randomness
- When may McNemar’s test be appropriate?
- For any continuous score without assumptions
- Only for latency measurements
- For comparing unrelated samples
- For paired binary outcomes on the same items
- What should be reported alongside a significance result?
- Only which model won
- Effect size, confidence interval, sample size, and test method
- Only a p-value
- A screenshot of the leaderboard
- Perché possono essere utili le esecuzioni ripetute del modello?
- They guarantee deterministic output
- They replace a representative test set
- They estimate run-to-run variability separately from variation across test examples
- They always increase the mean score
- La significatività statistica dimostra che un guadagno è importante per gli utenti?
- Yes, by definition
- No; practical effect size and product goals must also be considered
- Yes, if the p-value is small
- No, because statistics cannot inform decisions
- What should analysts consider when testing many models and metrics?
- Multiple-comparison risk or treating exploratory results as provisional
- Ignoring all confidence intervals
- Choosing only the largest score after testing
- Changing the test set after seeing results
- In che modo i team possono ridurre le decisioni di rilascio distorte a causa di un piccolo aumento del punteggio?
- Ship every statistically higher score
- Test only on examples used to tune the prompt
- Ignore subgroup changes
- Use a predeclared practical threshold and inspect representative held-out examples