引導式測驗 · 中等 等級

ML 模型測驗的 A/B 測試

檢查線上模型實驗的基礎,包括隨機分配、假設檢定、自適應分配和護欄。

問題 1 的 8

為什麼團隊在生產中進行 A/B 測試模型而不是信任離線指標?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. 為什麼團隊在生產中進行 A/B 測試模型而不是信任離線指標?
    • 具有更好離線準確性的模型仍然會損害真實的業務指標
    • 離線指標不合法
    • 生產比離線測試便宜
    • 離線資料不存在
  2. 隨機分配在 A/B 測試中扮演什麼角色?
    • 它使測試運行得更快
    • It supports balance between groups in expectation, subject to design and implementation assumptions
    • 它刪除有偏見的用戶
    • 它減少了所需模型的數量
  3. 在模型 A/B 測試中,治療組通常會發生什麼?
    • The candidate scores requests but all outputs are withheld
    • Every user is assigned to both models for the same decision
    • The candidate model serves user-facing decisions for the randomly assigned group
    • The control model is removed before any comparison
  4. 在原假設和規定的檢定假設下,如何解釋 p 值?
    • As the probability that the null hypothesis is true
    • As the probability that the treatment caused the observed result
    • As the probability that the experiment will replicate
    • As the probability of data at least this extreme if the null hypothesis holds
  5. 多臂老虎機如何在實驗運作時改變實驗?
    • It can allocate more traffic to variants with better observed outcomes and less to others
    • It keeps every variant at a fixed equal allocation throughout
    • It removes randomization from every assignment
    • It guarantees the best variant is identified without uncertainty
  6. A/B 測試前功率分析的目的為何?
    • 為實驗選擇一個徽標
    • 估計需要多少用戶才能可靠地偵測到真實效果
    • 加密流量
    • 刪除控制組
  7. What does a sample-ratio mismatch (SRM) signal during an A/B test?
    • The treatment model is necessarily better
    • The experiment has sufficient statistical power
    • Observed assignment counts differ materially from the configured allocation, which can indicate a data or randomization problem
    • The primary metric improved causally
  8. Why include guardrail metrics alongside an experiment’s primary success metric?
    • To guarantee the candidate wins on every metric
    • To replace the primary metric after results are observed
    • To avoid defining a decision rule before the test
    • To detect meaningful harms in other outcomes even when the primary metric improves