引導式測驗 · 硬 等級
評估 AI 代理測驗
測試代理評估如何對任務成功、軌跡和工具使用進行評分,為什麼基於環境的評分和重複試驗很重要,以及評分者的權衡。
問題 1 的 8
為什麼檢查單一最終答案通常對客服人員來說還不夠?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- 為什麼檢查單一最終答案通常對客服人員來說還不夠?
- Agents never produce answers
- Agents act over many steps and cause side effects, so the path and final state matter
- Answers are always correct
- Reference answers cannot be written for agent tasks
- SWE-bench 如何判斷代理是否解決了問題?
- By running the repository's tests after the agent's patch
- By counting lines changed
- By asking the agent if it succeeded
- By measuring response speed
- pass^k 風格指標測量什麼?
- Speed across k tools
- Success on at least one of k attempts
- Whether the agent succeeds on every one of k repeated attempts
- The number of tokens per step
- 哪個維度會標記一個成功但需要 50 步才能完成五步驟工作的代理?
- Task success
- 安全
- Tool naming
- Trajectory quality
- 基於模型的評分器的已知弱點是什麼?
- 他們無法閱讀文字
- They can be biased, for example toward long or confident answers, so they need checking against human labels
- They are always slower than humans
- 他們只處理圖像
- Why reset the environment to a known state before every run?
- So results are reproducible and runs cannot affect each other
- To save disk space
- Because agents require it to start
- To make tasks harder
- Why can public leaderboard scores mislead a team?
- They are always faked
- Leaderboards never publish numbers
- Benchmarks can leak into training data, saturate or fail to match your workload
- They only measure cost
- How should a safety eval treat a run where the agent finished the task but obeyed a planted instruction to leak data?
- As a full success
- As a partial success worth half credit
- It should be ignored
- As a failure