Guide-linked quiz · Hard Level

Asking LLMs How Confident They Are Quiz

Tests confidence calibration, ECE, study differences, token likelihoods, task-specific validation, and repeated-answer limits for consequential workflows.

Question 1 of 8

What does a model-generated “80% confident” statement establish by itself?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. What does a model-generated “80% confident” statement establish by itself?
    • That the answer is correct 80% of the time
    • Only that the model generated a confidence statement, not a calibrated probability
    • That it used retrieved sources
    • That it has a calibrated uncertainty module
  2. How should calibration of a confidence score be assessed?
    • By asking the same question to one model twice
    • By inspecting one answer
    • Across a relevant labeled set, comparing confidence levels with observed correctness
    • By checking whether the score is a round number
  3. What does the guide say about research on verbalized confidence?
    • It is always better than token likelihoods
    • It is universally calibrated on current systems
    • Every study finds it useless
    • Results vary by model, prompt, and evaluation data
  4. What does ECE summarize?
    • Mismatch between confidence and observed accuracy across confidence bins
    • The number of tokens in an answer
    • The quality of a model’s explanations
    • Whether each individual answer cites a source
  5. What do generated-token log probabilities directly describe?
    • The probability the whole answer is factually correct
    • Likelihood assigned to generated tokens, not directly correctness of the complete answer
    • The user’s confidence
    • Whether the answer is legally safe
  6. Before using confidence to route production tickets, what should a team do?
    • Remove human review
    • Set a threshold by intuition
    • Validate score behavior on labeled examples similar to production cases
    • Assume all categories calibrate equally
  7. Which limitation applies to repeated-answer agreement as a confidence signal?
    • It cannot be measured
    • It always reduces latency
    • Samples may share the same misconception
    • Agreement proves the answer is true
  8. In a consequential workflow, how should verbal confidence be used?
    • As an unverified automatic approval rule
    • Trust any answer above 50%
    • Replace external evidence with the score
    • As one validated signal alongside evidence and review

Keep testing yourself

More quizzes picked for your level.

View all quizzes