Guide-linked quiz · Hard Level
Asking LLMs How Confident They Are Quiz
Tests confidence calibration, ECE, study differences, token likelihoods, task-specific validation, and repeated-answer limits for consequential workflows.
Question 1 of 8
What does a model-generated “80% confident” statement establish by itself?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- What does a model-generated “80% confident” statement establish by itself?
- That the answer is correct 80% of the time
- Only that the model generated a confidence statement, not a calibrated probability
- That it used retrieved sources
- That it has a calibrated uncertainty module
- How should calibration of a confidence score be assessed?
- By asking the same question to one model twice
- By inspecting one answer
- Across a relevant labeled set, comparing confidence levels with observed correctness
- By checking whether the score is a round number
- What does the guide say about research on verbalized confidence?
- It is always better than token likelihoods
- It is universally calibrated on current systems
- Every study finds it useless
- Results vary by model, prompt, and evaluation data
- What does ECE summarize?
- Mismatch between confidence and observed accuracy across confidence bins
- The number of tokens in an answer
- The quality of a model’s explanations
- Whether each individual answer cites a source
- What do generated-token log probabilities directly describe?
- The probability the whole answer is factually correct
- Likelihood assigned to generated tokens, not directly correctness of the complete answer
- The user’s confidence
- Whether the answer is legally safe
- Before using confidence to route production tickets, what should a team do?
- Remove human review
- Set a threshold by intuition
- Validate score behavior on labeled examples similar to production cases
- Assume all categories calibrate equally
- Which limitation applies to repeated-answer agreement as a confidence signal?
- It cannot be measured
- It always reduces latency
- Samples may share the same misconception
- Agreement proves the answer is true
- In a consequential workflow, how should verbal confidence be used?
- As an unverified automatic approval rule
- Trust any answer above 50%
- Replace external evidence with the score
- As one validated signal alongside evidence and review
Keep testing yourself
More quizzes picked for your level.