Викторина, связанная с руководством · Средний Уровень
Анализ ошибок для викторины по приложениям LLM
Используйте репрезентативные трассировки, согласованные метки сбоев, уровень серьезности и последующее тестирование, чтобы расставить приоритеты исправлений приложений LLM.
Вопрос 1 из 8
Why do teams perform error analysis?
Предварительный просмотр каждого вопроса8 вопросы
Ответы и пояснения появляются по мере прохождения теста.
- Why do teams perform error analysis?
- To categorize and measure failures so teams can prioritize fixes
- To prove every model output is wrong
- To replace all evaluation datasets
- To select the most complex model
- Why review traces instead of only reading user complaints?
- Complaints always include the full root cause
- Trace review guarantees a fix
- Traces can show prompts, retrieval, tools, and application steps involved in a failure
- User reports are never useful
- What should be defined before labeling failures?
- The final fix
- A consistent taxonomy and clear outcome criteria
- Only a model name
- A positive user rating
- How should rare but high-severity errors be handled?
- Ignore them because they are uncommon
- Mix them into a random sample and hide severity
- Count every targeted example as representative frequency
- Use targeted sampling to inspect them, separately from prevalence estimates
- Why record model, prompt, and tool versions in error analysis?
- To connect failures to the system configuration that produced them
- To make labels more positive
- To avoid test cases
- To reveal user passwords
- What does measuring error frequency alone miss?
- The number of failures
- The sample size
- The taxonomy labels
- Severity, user impact, and subgroup differences
- After applying a fix, which check can reveal regressions?
- Delete all prior examples
- Score only the fixed example
- Rerun old failure cases and an independent holdout to check regressions
- Assume one prompt change works everywhere
- What limitation applies to a targeted failure sample?
- It cannot reveal any failure mode
- Its observed rate should not be treated as population prevalence
- It is always larger than a random sample
- It makes annotation unnecessary