Guide-linked quiz · Medium Level

What AI Confidence Scores Actually Mean Quiz

Explains calibration, population-level meaning, threshold choices, subgroup checks, and why a model's confidence score is not a guarantee.

Question 1 of 8

A calibrated binary classifier assigns 0.8 confidence to many cases. What should happen across that group over time?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. A calibrated binary classifier assigns 0.8 confidence to many cases. What should happen across that group over time?
    • About 80 percent should be positive, across a sufficiently large representative group
    • Every individual case must be correct
    • The model must abstain on every case
    • The score must equal accuracy on every subgroup
  2. What should you ask before interpreting a product's '90% confidence' badge?
    • Whether the number uses a large font
    • Whether the score is a calibrated probability and what outcome it predicts
    • Whether the product has a dark theme
    • Whether the model was written in Python
  3. Why should calibration data be separate from the model's training data?
    • To make every output equal 0.5
    • To remove the need for labels
    • To reduce overly optimistic estimates from examples the model already fit
    • To guarantee no future distribution shift
  4. A model is calibrated overall but not for one language group. What follows?
    • The overall score is sufficient for every user
    • The language group must be removed from evaluation
    • Calibration is irrelevant
    • The workflow should inspect subgroup outcomes and adjust use or calibration
  5. What does discrimination measure in this context?
    • How well the model ranks cases with different outcomes
    • Whether a score is displayed as a percentage
    • Whether a user can change the threshold
    • How fast the model runs
  6. When choosing an automation threshold, what should matter?
    • The number of decimal places in the score
    • The costs of false accepts and false rejects in the actual workflow
    • The model's training language only
    • Whether the threshold is a round number
  7. Why are token probabilities not automatically a factuality score for an entire generated answer?
    • They cannot be stored
    • They are always zero
    • They measure token likelihoods, not a validated probability that the response is correct
    • They only apply to images
  8. A low-confidence case could cause serious harm. Which workflow response is appropriate?
    • Hide the score and auto-approve it
    • Round the score upward
    • Assume calibration makes the case correct
    • Escalate or abstain according to a tested workflow

Keep testing yourself

More quizzes picked for your level.

View all quizzes