Útmutatóhoz kapcsolódó kvíz · Kemény Szint
Asking LLMs How Confident They Are Quiz
Tests confidence calibration, ECE, study differences, token likelihoods, task-specific validation, and repeated-answer limits for consequential workflows.
kérdés 1 -ból 8
What does a model-generated “80% confident” statement establish by itself?
Tekintse meg az összes kérdést8 kérdéseket
A válaszok és magyarázatok megjelennek a kvíz kitöltésekor.
- What does a model-generated “80% confident” statement establish by itself?
- That the answer is correct 80% of the time
- Only that the model generated a confidence statement, not a calibrated probability
- That it used retrieved sources
- That it has a calibrated uncertainty module
- How should calibration of a confidence score be assessed?
- By asking the same question to one model twice
- By inspecting one answer
- Across a relevant labeled set, comparing confidence levels with observed correctness
- By checking whether the score is a round number
- What does the guide say about research on verbalized confidence?
- It is always better than token likelihoods
- It is universally calibrated on current systems
- Every study finds it useless
- Results vary by model, prompt, and evaluation data
- What does ECE summarize?
- Mismatch between confidence and observed accuracy across confidence bins
- The number of tokens in an answer
- The quality of a model’s explanations
- Whether each individual answer cites a source
- What do generated-token log probabilities directly describe?
- The probability the whole answer is factually correct
- Likelihood assigned to generated tokens, not directly correctness of the complete answer
- The user’s confidence
- Whether the answer is legally safe
- Before using confidence to route production tickets, what should a team do?
- Remove human review
- Set a threshold by intuition
- Validate score behavior on labeled examples similar to production cases
- Assume all categories calibrate equally
- Which limitation applies to repeated-answer agreement as a confidence signal?
- Nem mérhető
- It always reduces latency
- Samples may share the same misconception
- Agreement proves the answer is true
- In a consequential workflow, how should verbal confidence be used?
- As an unverified automatic approval rule
- Trust any answer above 50%
- Replace external evidence with the score
- As one validated signal alongside evidence and review