Útmutatóhoz kapcsolódó kvíz · Kemény Szint

Az AI-ügynökök értékelése

Azt teszteli, hogy az ügynök hogyan értékeli a feladat sikerét, a pályákat és az eszközhasználatot, miért számítanak a környezetalapú osztályozásnak és az ismételt próbáknak, valamint az osztályozókkal kapcsolatos kompromisszumokat.

Kapcsolódó útmutató utakAz AI-ügynökök értékelése
kérdés 1 -ból 8

Why is checking a single final answer usually not enough for agents?

Tekintse meg az összes kérdést8 kérdéseket

A válaszok és magyarázatok megjelennek a kvíz kitöltésekor.

  1. Why is checking a single final answer usually not enough for agents?
    • Agents never produce answers
    • Agents act over many steps and cause side effects, so the path and final state matter
    • Answers are always correct
    • Reference answers cannot be written for agent tasks
  2. How does SWE-bench judge whether an agent solved an issue?
    • By running the repository's tests after the agent's patch
    • By counting lines changed
    • By asking the agent if it succeeded
    • By measuring response speed
  3. What does a pass^k style metric measure?
    • Speed across k tools
    • Success on at least one of k attempts
    • Whether the agent succeeds on every one of k repeated attempts
    • The number of tokens per step
  4. Which dimension would flag an agent that succeeds but takes fifty steps for a five-step job?
    • Task success
    • Biztonság
    • Tool naming
    • Trajectory quality
  5. What is a known weakness of model-based graders?
    • Nem tudnak szöveget olvasni
    • They can be biased, for example toward long or confident answers, so they need checking against human labels
    • They are always slower than humans
    • Csak képeken dolgoznak
  6. Why reset the environment to a known state before every run?
    • So results are reproducible and runs cannot affect each other
    • To save disk space
    • Because agents require it to start
    • To make tasks harder
  7. Why can public leaderboard scores mislead a team?
    • They are always faked
    • Leaderboards never publish numbers
    • Benchmarks can leak into training data, saturate or fail to match your workload
    • They only measure cost
  8. How should a safety eval treat a run where the agent finished the task but obeyed a planted instruction to leak data?
    • As a full success
    • As a partial success worth half credit
    • It should be ignored
    • As a failure