引導式測驗 · 硬 等級

評估 AI 代理測驗

測試代理評估如何對任務成功、軌跡和工具使用進行評分,為什麼基於環境的評分和重複試驗很重要,以及評分者的權衡。

相關引導路徑評估人工智慧代理
問題 1 的 8

為什麼檢查單一最終答案通常對客服人員來說還不夠?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. 為什麼檢查單一最終答案通常對客服人員來說還不夠?
    • Agents never produce answers
    • Agents act over many steps and cause side effects, so the path and final state matter
    • Answers are always correct
    • Reference answers cannot be written for agent tasks
  2. SWE-bench 如何判斷代理是否解決了問題?
    • By running the repository's tests after the agent's patch
    • By counting lines changed
    • By asking the agent if it succeeded
    • By measuring response speed
  3. pass^k 風格指標測量什麼?
    • Speed across k tools
    • Success on at least one of k attempts
    • Whether the agent succeeds on every one of k repeated attempts
    • The number of tokens per step
  4. 哪個維度會標記一個成功但需要 50 步才能完成五步驟工作的代理?
    • Task success
    • 安全
    • Tool naming
    • Trajectory quality
  5. 基於模型的評分器的已知弱點是什麼?
    • 他們無法閱讀文字
    • They can be biased, for example toward long or confident answers, so they need checking against human labels
    • They are always slower than humans
    • 他們只處理圖像
  6. Why reset the environment to a known state before every run?
    • So results are reproducible and runs cannot affect each other
    • To save disk space
    • Because agents require it to start
    • To make tasks harder
  7. Why can public leaderboard scores mislead a team?
    • They are always faked
    • Leaderboards never publish numbers
    • Benchmarks can leak into training data, saturate or fail to match your workload
    • They only measure cost
  8. How should a safety eval treat a run where the agent finished the task but obeyed a planted instruction to leak data?
    • As a full success
    • As a partial success worth half credit
    • It should be ignored
    • As a failure