GUIDE teknik

jàngat Agent IA

Evaluating AI agents means measuring how well a multi-step system reaches goals in realistic settings.

Déglul tegtal biiIA-nettali · ~3 min

Ci xët wii4 simili jàng
  1. Résumé
  2. Plongeur bu xóot
  3. Doxal ci àdduna dëgg
  4. The Future of Evaluating AI Agents
  5. Teg ko ci jëf
  6. Weyal di banneexu
  7. Laaj yi ñuy faral di laaj

Résumé

That covers whether it finished the task, whether the path it took was sensible, whether it used tools correctly, what it cost, and whether it stayed safe. Agents act over many steps and have side effects, so checking a single answer against a reference is not enough, and good evaluations usually run the agent inside a controlled environment and inspect the final state.

Plongeur bu xóot

Agent evaluation looks at several things. Task success asks whether the goal was actually achieved, ideally checked by looking at the environment afterward: tests pass, the right record exists, the file has the right contents. Trajectory quality asks whether the steps were reasonable, without loops, pointless calls or lucky guesses. Tool-use correctness checks whether the agent picked the right tools with valid arguments and handled errors. Cost and latency track tokens, calls and wall-clock time. Safety checks whether the agent avoided harmful or unauthorized actions and resisted prompt injection. Environment-based benchmarks have become a leading approach. SWE-bench, from Princeton researchers, asks agents to resolve real GitHub issues and checks the results with the project's tests, and SWE-bench Verified is a human-checked subset. WebArena provides self-hosted websites for browsing tasks. OSWorld tests agents operating full computer desktops. GAIA poses questions that need multi-step research and tools. The tau-bench suite simulates users and domain policies, and it introduced a pass^k metric that asks whether an agent succeeds on every one of k repeated attempts. Grading uses three main kinds of checks: code-based checks on outcomes, which are precise and cheap; model-based graders that score transcripts against a rubric, which are flexible but need to be checked against human judgment; and human review, which is the most trustworthy and the slowest. A common mistake is to trust public leaderboards alone. Benchmarks can leak into training data, become saturated, or fail to resemble your own workload. Another mistake is treating one run as the truth, since agents are nondeterministic and small differences in success rate can be noise. Teams get the most value from a private evaluation set built from their own real tasks and failures, run repeatedly and tracked over time.

Doxal ci àdduna dëgg

A coding-agent team scores each attempt by whether the repository's hidden tests pass after the agent's patch, as SWE-bench does, instead of judging whether the diff looks right.

A customer-service agent runs against simulated users and a mock booking database. Graders check whether the final database state matches the correct outcome and whether the agent followed refund policy.

An engineering team runs every task five times and reports how often the agent succeeds on all five, which exposes flaky behavior that a single-run success rate would hide.

A company adds a safety suite in which tasks include a planted instruction to delete files or leak data, and it counts an otherwise successful run as failed if the agent obeyed.

The Future of Evaluating AI Agents

As agents take on longer tasks, evaluations are moving toward longer horizons, more realistic environments, and measures of reliability and cost as well as peak capability. Benchmark saturation and contamination will probably keep pushing developers to create new tasks and to rely more on private, domain-specific test sets. Safety and security evaluations, including resistance to prompt injection, are receiving more attention as agents get access to real systems. There is no agreed standard for grading open-ended agent behavior, so combining outcome checks, calibrated model graders and human review is likely to remain common practice.

Teg ko ci jëf

Lu tax mu am solo

  • Njëgg ak budget. Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
  • dogal yu gëna leer. Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
  • Xool kalite. Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.

Moytul

  • Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
  • Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
  • Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.

Jéego yi ci topp

  1. Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
  2. Benchmark ci biir sargal ak done yu dëggu.
  3. Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
  4. Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.

Weyal di banneexu

Bulletin bu amul fayda

Jàngal xibaar AI bu bes

Ñatti jaar-jaari IA yuñ firndeel bis bu nekk ci ayu-bis bi, ñu bind leen ci Àngle bu leer. Gratis ba fàww, amul yëgle.

Benn email bis bu nekk ci ayu-bis bi. Benn klik nga dindi abonemaa bi. Dunu musa jaay wala séddoo sa adres.

Nattal sa bopp

Take the Evaluating AI Agents quiz

Feedback ci saasi ci bépp tontu, ak sertifikaa buñ mëna séddoo ak ID buñ mëna saytu soo jàllee ci njàng mi.

Tambalil quiz

Jàppale njàngum IA te doo fay. AI Understanding mbootaay la buy def te Yàlla tax — amul yëgle, amul paywall, musul am. Defal ab sarax

Laaj yi ñuy faral di laaj

What is Evaluating AI Agents?

Evaluating AI agents means measuring how well a multi-step system reaches goals in realistic settings. That covers whether it finished the task, whether the path it took was sensible, whether it used tools correctly, what it cost, and whether it stayed safe. Agents act over many steps and have side effects, so checking a single answer against a reference is not enough, and good evaluations usually run the agent inside a controlled environment and inspect the final state.

Why is checking a single final answer usually not enough for agents?

Multi-step actions change environments, so you need to check what actually happened and how, not just the final text.

How does SWE-bench judge whether an agent solved an issue?

Environment-based checks, here the project's tests, judge whether the fix really works.

What does a pass^k style metric measure?

Requiring success on all k attempts measures consistency, which matters for agents deployed to real users.

Which dimension would flag an agent that succeeds but takes fifty steps for a five-step job?

Trajectory checks judge the path taken, catching loops and waste that outcome checks alone miss.

What is a known weakness of model-based graders?

Model graders are flexible but should be checked against human judgment and watched for systematic bias.