Викторина, связанная с руководством · Средний Уровень
Что на самом деле означают показатели уверенности ИИ
Объясняет калибровку, значение на уровне совокупности, выбор пороговых значений, проверки подгрупп и почему показатель достоверности модели не является гарантией.
Вопрос 1 из 8
A calibrated binary classifier assigns 0.8 confidence to many cases. What should happen across that group over time?
Предварительный просмотр каждого вопроса8 вопросы
Ответы и пояснения появляются по мере прохождения теста.
- A calibrated binary classifier assigns 0.8 confidence to many cases. What should happen across that group over time?
- About 80 percent should be positive, across a sufficiently large representative group
- Every individual case must be correct
- The model must abstain on every case
- The score must equal accuracy on every subgroup
- What should you ask before interpreting a product's '90% confidence' badge?
- Whether the number uses a large font
- Whether the score is a calibrated probability and what outcome it predicts
- Whether the product has a dark theme
- Whether the model was written in Python
- Why should calibration data be separate from the model's training data?
- To make every output equal 0.5
- To remove the need for labels
- To reduce overly optimistic estimates from examples the model already fit
- To guarantee no future distribution shift
- A model is calibrated overall but not for one language group. What follows?
- The overall score is sufficient for every user
- The language group must be removed from evaluation
- Calibration is irrelevant
- The workflow should inspect subgroup outcomes and adjust use or calibration
- What does discrimination measure in this context?
- How well the model ranks cases with different outcomes
- Whether a score is displayed as a percentage
- Whether a user can change the threshold
- Как быстро работает модель
- When choosing an automation threshold, what should matter?
- The number of decimal places in the score
- The costs of false accepts and false rejects in the actual workflow
- The model's training language only
- Whether the threshold is a round number
- Why are token probabilities not automatically a factuality score for an entire generated answer?
- They cannot be stored
- Они всегда равны нулю
- They measure token likelihoods, not a validated probability that the response is correct
- They only apply to images
- A low-confidence case could cause serious harm. Which workflow response is appropriate?
- Hide the score and auto-approve it
- Round the score upward
- Assume calibration makes the case correct
- Escalate or abstain according to a tested workflow