Kvíz spojený s průvodcem · Těžko úroveň

Asking LLMs How Confident They Are Quiz

Tests confidence calibration, ECE, study differences, token likelihoods, task-specific validation, and repeated-answer limits for consequential workflows.

Související vodicí cestyVerbalized Confidence From LLMs
Otázka 1 z 8

Co samo o sobě stanoví modelem vygenerované prohlášení „80% sebevědomí“?

Náhled každé otázky8 otázky

Odpovědi a vysvětlení se zobrazují, jak budete v kvízu.

  1. Co samo o sobě stanoví modelem vygenerované prohlášení „80% sebevědomí“?
    • That the answer is correct 80% of the time
    • Only that the model generated a confidence statement, not a calibrated probability
    • That it used retrieved sources
    • That it has a calibrated uncertainty module
  2. Jak by se měla hodnotit kalibrace skóre spolehlivosti?
    • By asking the same question to one model twice
    • By inspecting one answer
    • Across a relevant labeled set, comparing confidence levels with observed correctness
    • By checking whether the score is a round number
  3. Co průvodce říká o výzkumu verbalizované sebedůvěry?
    • It is always better than token likelihoods
    • It is universally calibrated on current systems
    • Every study finds it useless
    • Results vary by model, prompt, and evaluation data
  4. Co shrnuje ECE?
    • Mismatch between confidence and observed accuracy across confidence bins
    • The number of tokens in an answer
    • The quality of a model’s explanations
    • Whether each individual answer cites a source
  5. Co přímo popisují pravděpodobnosti protokolu generovaných tokenů?
    • The probability the whole answer is factually correct
    • Likelihood assigned to generated tokens, not directly correctness of the complete answer
    • The user’s confidence
    • Whether the answer is legally safe
  6. Before using confidence to route production tickets, what should a team do?
    • Remove human review
    • Set a threshold by intuition
    • Validate score behavior on labeled examples similar to production cases
    • Assume all categories calibrate equally
  7. Which limitation applies to repeated-answer agreement as a confidence signal?
    • Nelze to změřit
    • It always reduces latency
    • Samples may share the same misconception
    • Agreement proves the answer is true
  8. In a consequential workflow, how should verbal confidence be used?
    • As an unverified automatic approval rule
    • Trust any answer above 50%
    • Replace external evidence with the score
    • As one validated signal alongside evidence and review