Викторина, связанная с руководством · Средний Уровень

Тест на оценку онлайн- и офлайн-моделей

Отличайте эффективность исторических данных от реального воздействия и определяйте предположения и меры предосторожности при автономном воспроизведении и онлайн-экспериментах.

Связанные направляющие путиОнлайн и оффлайн оценка модели
Вопрос 1 из 8

What does offline evaluation directly measure?

Предварительный просмотр каждого вопроса8 вопросы

Ответы и пояснения появляются по мере прохождения теста.

  1. What does offline evaluation directly measure?
    • Candidate metrics on a specified historical or held-out dataset
    • Causal impact for every future user
    • The complete product effect in production
    • A guarantee of fairness
  2. Why can historical recommender logs bias offline evaluation?
    • Every candidate was shown equally
    • Prior exposure determined which items could receive interactions
    • Click labels are always complete
    • Offline scores include live randomization
  3. What can a properly randomized online experiment estimate?
    • A model's exact causal mechanism
    • Universal performance for every population
    • The average effect of assignment for the eligible tested population under design assumptions
    • The offline test-set loss
  4. Why run offline checks before online exposure?
    • They ensure historical labels are unbiased
    • They replace all live testing
    • They guarantee no guardrail regression
    • They can catch broken or clearly poor candidates with lower user risk
  5. Which rollout pattern limits exposure while measuring a candidate live?
    • Guaranteeing randomized assignment
    • Limiting exposure while monitoring a candidate's live behavior
    • Measuring only historical accuracy
    • Removing the need for rollback
  6. Why predeclare primary and guardrail metrics?
    • It makes every metric statistically independent
    • It guarantees enough sample size
    • It reduces post-hoc selection and clarifies success and harm criteria
    • It removes the need for experiment logs
  7. What can offline-online disagreement reveal?
    • That the model is causal
    • That the online test must be false
    • That offline data are useless
    • Distribution, logging, metric or product-interaction mismatch
  8. Why does an online experiment not guarantee impact in every future context?
    • The result is tied to its tested population, period and conditions
    • Randomization is never useful
    • The experiment measures only latency
    • Online tests cannot record assignment