引導式測驗 · 硬 等級
詢問法學碩士的信心程度測驗
測試後續工作流程的置信度校準、ECE、研究差異、標記可能性、特定任務的驗證和重複答案限制。
問題 1 的 8
What does a model-generated “80% confident” statement establish by itself?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- What does a model-generated “80% confident” statement establish by itself?
- That the answer is correct 80% of the time
- Only that the model generated a confidence statement, not a calibrated probability
- That it used retrieved sources
- That it has a calibrated uncertainty module
- How should calibration of a confidence score be assessed?
- By asking the same question to one model twice
- By inspecting one answer
- Across a relevant labeled set, comparing confidence levels with observed correctness
- By checking whether the score is a round number
- What does the guide say about research on verbalized confidence?
- It is always better than token likelihoods
- It is universally calibrated on current systems
- Every study finds it useless
- Results vary by model, prompt, and evaluation data
- What does ECE summarize?
- Mismatch between confidence and observed accuracy across confidence bins
- The number of tokens in an answer
- The quality of a model’s explanations
- Whether each individual answer cites a source
- What do generated-token log probabilities directly describe?
- The probability the whole answer is factually correct
- Likelihood assigned to generated tokens, not directly correctness of the complete answer
- The user’s confidence
- Whether the answer is legally safe
- Before using confidence to route production tickets, what should a team do?
- Remove human review
- Set a threshold by intuition
- Validate score behavior on labeled examples similar to production cases
- Assume all categories calibrate equally
- Which limitation applies to repeated-answer agreement as a confidence signal?
- 無法測量
- It always reduces latency
- Samples may share the same misconception
- Agreement proves the answer is true
- In a consequential workflow, how should verbal confidence be used?
- As an unverified automatic approval rule
- Trust any answer above 50%
- Replace external evidence with the score
- As one validated signal alongside evidence and review