Ikibazo gihuza ibibazo · Hagati Urwego

What AI Confidence Scores Actually Mean Quiz

Explains calibration, population-level meaning, threshold choices, subgroup checks, and why a model's confidence score is not a guarantee.

Inzira zijyanye no kuyoboraWhat AI Confidence Scores Mean
Ikibazo 1 Bya 8

Calibrated binary classifier itanga 0.8 ikizere kubibazo byinshi. Ni iki kigomba kubaho muri iryo tsinda mugihe runaka?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Calibrated binary classifier itanga 0.8 ikizere kubibazo byinshi. Ni iki kigomba kubaho muri iryo tsinda mugihe runaka?
    • About 80 percent should be positive, across a sufficiently large representative group
    • Every individual case must be correct
    • The model must abstain on every case
    • The score must equal accuracy on every subgroup
  2. Niki ukwiye kubaza mbere yo gusobanura ikirango cyibicuruzwa '90%?'
    • Whether the number uses a large font
    • Whether the score is a calibrated probability and what outcome it predicts
    • Whether the product has a dark theme
    • Whether the model was written in Python
  3. Kuki kalibrasi igomba gutandukana namakuru yimyitozo yicyitegererezo?
    • To make every output equal 0.5
    • To remove the need for labels
    • To reduce overly optimistic estimates from examples the model already fit
    • To guarantee no future distribution shift
  4. Icyitegererezo kirahinduka muri rusange ariko ntabwo ari itsinda ryururimi rumwe. Ni iki gikurikira?
    • The overall score is sufficient for every user
    • The language group must be removed from evaluation
    • Calibration is irrelevant
    • The workflow should inspect subgroup outcomes and adjust use or calibration
  5. Ivangura ripima iki muri uru rwego?
    • How well the model ranks cases with different outcomes
    • Whether a score is displayed as a percentage
    • Whether a user can change the threshold
    • Ukuntu icyitegererezo cyihuta
  6. When choosing an automation threshold, what should matter?
    • The number of decimal places in the score
    • The costs of false accepts and false rejects in the actual workflow
    • The model's training language only
    • Whether the threshold is a round number
  7. Why are token probabilities not automatically a factuality score for an entire generated answer?
    • They cannot be stored
    • They are always zero
    • They measure token likelihoods, not a validated probability that the response is correct
    • They only apply to images
  8. A low-confidence case could cause serious harm. Which workflow response is appropriate?
    • Hide the score and auto-approve it
    • Round the score upward
    • Assume calibration makes the case correct
    • Escalate or abstain according to a tested workflow