Quiz collegato alla guida · Medio Livello

Statistical Significance in LLM Evals Quiz

Interpret evaluation score differences using matched test design, uncertainty intervals, effect size, and practical relevance for product decisions.

Domanda 1 di 8

Cosa valuta un test di significatività statistica?

Anteprima di ogni domanda8 domande

Le risposte e le spiegazioni vengono visualizzate mentre rispondi al quiz.

  1. Cosa valuta un test di significatività statistica?
    • Whether a model is safe in every deployment
    • Whether users prefer a new interface
    • Whether an observed result would be surprising under a specified null hypothesis
    • Whether a score increase is large enough to matter
  2. Perché utilizzare un confronto appaiato quando entrambi i sistemi rispondono agli stessi elementi di valutazione?
    • It preserves item-level correspondence when estimating score differences
    • It guarantees the systems have identical scores
    • It makes every example independent
    • It removes all output randomness
  3. When may McNemar’s test be appropriate?
    • For any continuous score without assumptions
    • Only for latency measurements
    • For comparing unrelated samples
    • For paired binary outcomes on the same items
  4. What should be reported alongside a significance result?
    • Only which model won
    • Effect size, confidence interval, sample size, and test method
    • Only a p-value
    • A screenshot of the leaderboard
  5. Perché possono essere utili le esecuzioni ripetute del modello?
    • They guarantee deterministic output
    • They replace a representative test set
    • They estimate run-to-run variability separately from variation across test examples
    • They always increase the mean score
  6. La significatività statistica dimostra che un guadagno è importante per gli utenti?
    • Yes, by definition
    • No; practical effect size and product goals must also be considered
    • Yes, if the p-value is small
    • No, because statistics cannot inform decisions
  7. What should analysts consider when testing many models and metrics?
    • Multiple-comparison risk or treating exploratory results as provisional
    • Ignoring all confidence intervals
    • Choosing only the largest score after testing
    • Changing the test set after seeing results
  8. In che modo i team possono ridurre le decisioni di rilascio distorte a causa di un piccolo aumento del punteggio?
    • Ship every statistically higher score
    • Test only on examples used to tune the prompt
    • Ignore subgroup changes
    • Use a predeclared practical threshold and inspect representative held-out examples