引導式測驗 · 中等 等級
ML 模型測驗的 A/B 測試
檢查線上模型實驗的基礎,包括隨機分配、假設檢定、自適應分配和護欄。
問題 1 的 8
為什麼團隊在生產中進行 A/B 測試模型而不是信任離線指標?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- 為什麼團隊在生產中進行 A/B 測試模型而不是信任離線指標?
- 具有更好離線準確性的模型仍然會損害真實的業務指標
- 離線指標不合法
- 生產比離線測試便宜
- 離線資料不存在
- 隨機分配在 A/B 測試中扮演什麼角色?
- 它使測試運行得更快
- It supports balance between groups in expectation, subject to design and implementation assumptions
- 它刪除有偏見的用戶
- 它減少了所需模型的數量
- 在模型 A/B 測試中,治療組通常會發生什麼?
- The candidate scores requests but all outputs are withheld
- Every user is assigned to both models for the same decision
- The candidate model serves user-facing decisions for the randomly assigned group
- The control model is removed before any comparison
- 在原假設和規定的檢定假設下,如何解釋 p 值?
- As the probability that the null hypothesis is true
- As the probability that the treatment caused the observed result
- As the probability that the experiment will replicate
- As the probability of data at least this extreme if the null hypothesis holds
- 多臂老虎機如何在實驗運作時改變實驗?
- It can allocate more traffic to variants with better observed outcomes and less to others
- It keeps every variant at a fixed equal allocation throughout
- It removes randomization from every assignment
- It guarantees the best variant is identified without uncertainty
- A/B 測試前功率分析的目的為何?
- 為實驗選擇一個徽標
- 估計需要多少用戶才能可靠地偵測到真實效果
- 加密流量
- 刪除控制組
- What does a sample-ratio mismatch (SRM) signal during an A/B test?
- The treatment model is necessarily better
- The experiment has sufficient statistical power
- Observed assignment counts differ materially from the configured allocation, which can indicate a data or randomization problem
- The primary metric improved causally
- Why include guardrail metrics alongside an experiment’s primary success metric?
- To guarantee the candidate wins on every metric
- To replace the primary metric after results are observed
- To avoid defining a decision rule before the test
- To detect meaningful harms in other outcomes even when the primary metric improves