Викторина, связанная с руководством · Жесткий Уровень

Как спросить у выпускников LLM, насколько они уверены в себе? Тест

Тестирует калибровку достоверности, ECE, различия в исследованиях, вероятность токенов, проверку для конкретных задач и пределы повторных ответов для последующих рабочих процессов.

Связанные направляющие путиВербализованная уверенность со стороны LLM
Вопрос 1 из 8

What does a model-generated “80% confident” statement establish by itself?

Предварительный просмотр каждого вопроса8 вопросы

Ответы и пояснения появляются по мере прохождения теста.

  1. What does a model-generated “80% confident” statement establish by itself?
    • That the answer is correct 80% of the time
    • Only that the model generated a confidence statement, not a calibrated probability
    • That it used retrieved sources
    • That it has a calibrated uncertainty module
  2. How should calibration of a confidence score be assessed?
    • By asking the same question to one model twice
    • By inspecting one answer
    • Across a relevant labeled set, comparing confidence levels with observed correctness
    • By checking whether the score is a round number
  3. What does the guide say about research on verbalized confidence?
    • It is always better than token likelihoods
    • It is universally calibrated on current systems
    • Every study finds it useless
    • Results vary by model, prompt, and evaluation data
  4. What does ECE summarize?
    • Mismatch between confidence and observed accuracy across confidence bins
    • The number of tokens in an answer
    • The quality of a model’s explanations
    • Whether each individual answer cites a source
  5. What do generated-token log probabilities directly describe?
    • The probability the whole answer is factually correct
    • Likelihood assigned to generated tokens, not directly correctness of the complete answer
    • The user’s confidence
    • Whether the answer is legally safe
  6. Before using confidence to route production tickets, what should a team do?
    • Remove human review
    • Set a threshold by intuition
    • Validate score behavior on labeled examples similar to production cases
    • Assume all categories calibrate equally
  7. Which limitation applies to repeated-answer agreement as a confidence signal?
    • Это невозможно измерить
    • It always reduces latency
    • Samples may share the same misconception
    • Agreement proves the answer is true
  8. In a consequential workflow, how should verbal confidence be used?
    • As an unverified automatic approval rule
    • Trust any answer above 50%
    • Replace external evidence with the score
    • As one validated signal alongside evidence and review