Quiz collegato alla guida · Difficile Livello

Quiz sulla valutazione degli agenti IA

Verifica il modo in cui gli agenti valutano il successo delle attività, le traiettorie e l'uso degli strumenti, il motivo per cui la valutazione basata sull'ambiente e le prove ripetute sono importanti e i compromessi del valutatore.

Percorsi guida correlatiValutazione degli agenti IA
Domanda 1 di 8

Why is checking a single final answer usually not enough for agents?

Anteprima di ogni domanda8 domande

Le risposte e le spiegazioni vengono visualizzate mentre rispondi al quiz.

  1. Why is checking a single final answer usually not enough for agents?
    • Agents never produce answers
    • Agents act over many steps and cause side effects, so the path and final state matter
    • Answers are always correct
    • Reference answers cannot be written for agent tasks
  2. How does SWE-bench judge whether an agent solved an issue?
    • By running the repository's tests after the agent's patch
    • By counting lines changed
    • By asking the agent if it succeeded
    • By measuring response speed
  3. What does a pass^k style metric measure?
    • Speed across k tools
    • Success on at least one of k attempts
    • Whether the agent succeeds on every one of k repeated attempts
    • The number of tokens per step
  4. Which dimension would flag an agent that succeeds but takes fifty steps for a five-step job?
    • Task success
    • Safety
    • Tool naming
    • Trajectory quality
  5. What is a known weakness of model-based graders?
    • Non possono leggere il testo
    • They can be biased, for example toward long or confident answers, so they need checking against human labels
    • They are always slower than humans
    • Funzionano solo sulle immagini
  6. Why reset the environment to a known state before every run?
    • So results are reproducible and runs cannot affect each other
    • To save disk space
    • Because agents require it to start
    • To make tasks harder
  7. Why can public leaderboard scores mislead a team?
    • They are always faked
    • Leaderboards never publish numbers
    • Benchmarks can leak into training data, saturate or fail to match your workload
    • They only measure cost
  8. How should a safety eval treat a run where the agent finished the task but obeyed a planted instruction to leak data?
    • As a full success
    • As a partial success worth half credit
    • It should be ignored
    • As a failure