quiz bu lëkkale ak tegtal · Dëgër Niveau

Jàngat IA Agents Quiz

Dafay saytu ni agent evals yi di joxee poñ ci liggéey bi, trajectoire yi ak jëfandikoo jumtukaay yi, lu tax grade bu sukkandiko ci environmaa bi ak jéem yu bari dañu am solo, ak grade-offs yi.

Yooni njiit yu jëm cijàngat Agent IA
Laaj 1 de 8

Why is checking a single final answer usually not enough for agents?

Xoolaatal laaj bu nekk8 laaj

Sooy def quiz bi, tontu yi ak leeral yi dañuy feeñ.

  1. Why is checking a single final answer usually not enough for agents?
    • Agents never produce answers
    • Agents act over many steps and cause side effects, so the path and final state matter
    • Answers are always correct
    • Reference answers cannot be written for agent tasks
  2. How does SWE-bench judge whether an agent solved an issue?
    • By running the repository's tests after the agent's patch
    • By counting lines changed
    • By asking the agent if it succeeded
    • By measuring response speed
  3. What does a pass^k style metric measure?
    • Speed across k tools
    • Success on at least one of k attempts
    • Whether the agent succeeds on every one of k repeated attempts
    • The number of tokens per step
  4. Which dimension would flag an agent that succeeds but takes fifty steps for a five-step job?
    • Task success
    • Safety
    • Tool naming
    • Trajectory quality
  5. What is a known weakness of model-based graders?
    • Mënu ñu jàng mbind
    • They can be biased, for example toward long or confident answers, so they need checking against human labels
    • They are always slower than humans
    • Ci nataal kese lañuy liggéey
  6. Why reset the environment to a known state before every run?
    • So results are reproducible and runs cannot affect each other
    • To save disk space
    • Because agents require it to start
    • To make tasks harder
  7. Why can public leaderboard scores mislead a team?
    • They are always faked
    • Leaderboards never publish numbers
    • Benchmarks can leak into training data, saturate or fail to match your workload
    • They only measure cost
  8. How should a safety eval treat a run where the agent finished the task but obeyed a planted instruction to leak data?
    • As a full success
    • As a partial success worth half credit
    • It should be ignored
    • As a failure