Kuiz berkaitan panduan · Keras Tahap

Menilai Kuiz Agen AI

Menguji cara evals ejen menjaringkan kejayaan tugasan, trajektori dan penggunaan alat, sebab penggredan berasaskan persekitaran dan ujian berulang penting, dan pertukaran penggred.

Laluan panduan yang berkaitanMenilai Agen AI
soalan 1 daripada 8

Why is checking a single final answer usually not enough for agents?

Pratonton setiap soalan8 soalan

Jawapan dan penjelasan muncul semasa anda mengambil kuiz.

  1. Why is checking a single final answer usually not enough for agents?
    • Agents never produce answers
    • Agents act over many steps and cause side effects, so the path and final state matter
    • Answers are always correct
    • Reference answers cannot be written for agent tasks
  2. How does SWE-bench judge whether an agent solved an issue?
    • By running the repository's tests after the agent's patch
    • By counting lines changed
    • By asking the agent if it succeeded
    • By measuring response speed
  3. What does a pass^k style metric measure?
    • Speed across k tools
    • Success on at least one of k attempts
    • Whether the agent succeeds on every one of k repeated attempts
    • The number of tokens per step
  4. Which dimension would flag an agent that succeeds but takes fifty steps for a five-step job?
    • Task success
    • Keselamatan
    • Tool naming
    • Trajectory quality
  5. What is a known weakness of model-based graders?
    • Mereka tidak boleh membaca teks
    • They can be biased, for example toward long or confident answers, so they need checking against human labels
    • They are always slower than humans
    • Mereka hanya berfungsi pada imej
  6. Why reset the environment to a known state before every run?
    • So results are reproducible and runs cannot affect each other
    • To save disk space
    • Because agents require it to start
    • To make tasks harder
  7. Why can public leaderboard scores mislead a team?
    • They are always faked
    • Leaderboards never publish numbers
    • Benchmarks can leak into training data, saturate or fail to match your workload
    • They only measure cost
  8. How should a safety eval treat a run where the agent finished the task but obeyed a planted instruction to leak data?
    • As a full success
    • As a partial success worth half credit
    • It should be ignored
    • As a failure