Subira ku makuru
Guhanga udushyaAI Understanding ibisobanuro

Benchmark isanga abakozi ba coding baharanira kubaka abakozi ba serivise yisi

Ibipimo bishya bisuzuma niba abakozi ba code bashobora kubaka serivisi zuzuye zabakiriya-serivisi zidafite ishingiro. Uru rupapuro ruvuga ko ibizamini bikomeye byageragejwe byatsinze 23.9% by'ibigereranyo, ugereranije na 82.2% ku mpuguke zabiherewe uburenganzira.

4 min readRead the primary source
Source-provided image accompanying Benchmark finds coding agents struggle to build real-world service agents
Inyandiko y'ibanzeInkomoko yanditse
Umwanditsi
arxiv.org
Ihuza ry'inkomoko
arxiv.orghttps://arxiv.org/abs/2609.04611
Ubwoko bw'inkomoko
Inyandiko y'ibanze - itangazo ryemewe, impapuro, dosiye, cyangwa urupapuro rwambere-dusoma mu buryo butaziguye.
ImirongoSobanukirwa ibi mumasegonda 60

Tangira hano

Amagambo y'ingenzi

Ibipimo
Ikizamini gisanzwe cyangwa dataset ikoreshwa mugupima no kugereranya imikorere yicyitegererezo.
API (Imigaragarire ya Porogaramu)
Inzira yuburyo bwa sisitemu imwe yohereza ibyifuzo no kwakira ibisubizo bivuye murindi sisitemu.
IsuzumeIkibazo cya AI

Byagenze bite

Abashakashatsi berekanye τ ^ τ-Bench, igipimo cyerekana ubwubatsi bwa nyuma kugeza ku ndunduro umurimo wa sisitemu ya code ya AI. Ibipimo biha abaterankunga ibikorwa byubucuruzi, ibisabwa kubakiriya, umusaruro wa API, codebase ihari, hamwe nimbibi kuri moderi hamwe nigiciro cyo gutanga, hanyuma ugasuzuma umukozi-serivisi waturutse kubakoresha bigana.

Inkomoko ya arXiv, yatanzwe ku ya 4 Nzeri 2026, isobanura τ ^ τ-Intebe nk'ahantu ho gusuzuma sisitemu ya AI yubaka abakozi aho gusubiza gusa ibipimo ngenderwaho. Umukozi ushinzwe iterambere yakira inyandiko zerekana ko ubucuruzi bugumana mubyukuri, ibisabwa kubakiriya, umusaruro API unyuramo ibikorwa bigomba gukora, codebase yarazwe, nimbogamizi zo gutanga ikiguzi no guhitamo icyitegererezo. Igomba gukoresha ibyo bikoresho kugirango itange abakiriya-serivisi yuzuye.

Umukozi wavuyemo asuzumwa no kubikoresha kubakoresha bigereranijwe. Hafi yimirimo 53 muri domaine enye, impapuro zivuga ko iboneza ryayo rikomeye-Claude Opus 5 yakoreshejwe binyuze muri Claude Code - yatsinze 23.9% yikigereranyo cyo gusuzuma. Impuguke zemewe n’impapuro zerekana amanota 82.2%. Ibi ni ibisubizo byatangajwe nimpapuro; inkomoko ntabwo itanga ibyemezo byigenga kurupapuro rwa arXiv.

Abanditsi bagaragaza uburyo bwo gutsindwa bavuga ko bisa nkibibazo byahuye nabateza imbere abakozi: ibibazo bidahwitse aho gusobanukirwa byimazeyo inyandiko zubucuruzi, itumanaho rito hamwe nabakiriya, hamwe nubushakashatsi budahagije hamwe nubwubatsi bwabakozi cyangwa gukoresha amafaranga yakoreshejwe. Baranga igipimo nkugerageza gukora umukozi wa koperative yubaka intego igaragara kubakozi ba code.

Ibisobanuro birambuye: arxiv.org ↗

Impamvu ari ngombwa

Ibipimo byerekana icyuho kiri mu isuzuma ryubu: niba sisitemu ya AI ishobora gutanga umukozi ukoreshwa mu mbogamizi ziterwa no gusezerana kwabakiriya nyabo. Ikinyuranyo cyerekana imikorere hagati yimiterere ikomeye yageragejwe hamwe ninzobere zerekana ko ubushobozi bwa code bwonyine budashyiraho ubushobozi bwo gusobanukirwa amakuru yubucuruzi, kuvugana nabakiriya, guhitamo imyubakire, cyangwa gucunga ibiciro.

Isuzuma ryinshi ritandukanya ubushobozi bwikitegererezo cyo kubyara code cyangwa kurangiza umurimo ugufi. . Ibyo bituma icyuho kivugwa gishobora kuba ingirakamaro kumatsinda ahitamo umubare wubwubatsi bwabantu no gusuzuma ibikorwa byubaka abakozi bikeneye.

Ibisubizo biratanga kandi uburyo bunoze bwo gusuzuma ibirego bivuga ko abakozi ba code bashobora gusimbuza cyangwa gukoresha cyane ibikorwa-byo gutangiza software-sisitemu ya AI. Nk’uko inkomoko ibivuga, sisitemu zapimwe akenshi zabyaye ikintu cyakoraga ariko nticyuzuze ibisabwa byimbitse byo gusezerana. Ibipimo rero bihindura ibitekerezo kuva kubisekuruza byonyine byerekeza kumiterere ya sisitemu yoherejwe hamwe n'imikoranire yabakoresha.

Igisubizo gikomeza kugarukira. Urupapuro rwurupapuro ntirurambuye kubijyanye nigipimo cyibikorwa byubaka, igishushanyo mbonera, uburyo bwo gutanga amanota, guhitamo ibyingenzi, cyangwa kutamenya neza imibare. Ntabwo kandi yerekana ko amanota 23.9% ateganya umusaruro. Uru rupapuro ni inyandiko ya arXiv, bityo ibyifuzo byayo bigomba gufatwa nkibyavuye mu bushakashatsi mu gihe hagitegerejwe gusuzumwa no kwigana.

Interactive Mechanism

Uburyo bukoreshwa: Uburyo bukora

Shakisha ikoranabuhanga ryihishe inyuma yiri terambere.

Agent Lifecycle Stage:
1
User Intent & Planning: "Audit customer refund request #4092 and settle payment."
2
Tool Calling: Emits structured JSON call crm_get_transaction(id='4092').
3
Guardrail & Verification:🛡️ Paused: High-value action requires human operator sign-off.
4
Final Settlement: Refund recorded, email receipt dispatched, and audit log stored.
Core takeaway: An AI agent is not just a language model—it is a closed loop of planning, tool invocation, and environment feedback. Production systems require self-healing retries and strict human approval guardrails.
Kugenzura Ibitekerezo Byagenzuwe+10 Points
AI Agents Quiz

An agent must create a draft calendar event for Tuesday at 2 p.m. Which evidence would establish the requested result?

Ibyo kureba

Urupapuro ntirugaragaza uburyo τ ^ τ-Intebe igereranya nibindi bipimo, niba abakoresha bayo bigana bahanura imikorere hamwe nabakiriya nyabo, cyangwa niba ibisubizo muri rusange birenze imirimo 53 yatangajwe na domaine enye. Inkomoko kandi ntabwo yerekana ibipimo ngenderwaho rusange, ibisabwa mubikorwa, ibiciro, cyangwa kwigana byigenga.

Niba abanditsi barekura ibipimo, ibisobanuro byakazi, ibikoresho byo gusuzuma, hamwe nibishyirwa mubikorwa nibyingenzi kubyara. Urupapuro rwinkomoko rwemeza impapuro no guhuza verisiyo ya PDF na HTML, ariko ntiruvuga ko igipimo ubwacyo gishobora kugerwaho kumugaragaro cyangwa kwerekana uburyo ubwo aribwo bwose cyangwa igiciro.

Isuzuma ry'ejo hazaza rigomba kugerageza izindi moderi, sisitemu ya coding-agent, domaine, hamwe nubwoko bwimirimo, mugihe usobanura uburyo abakoresha bigana bahagarariye imyitwarire nyayo yabakiriya. Kugereranya numukozi uriho, coding, hamwe na software-injeniyeri ngenderwaho byafasha kumenya ubushobozi bwinyongera τ ^ τ-Inteko.

Imipaka ivugwa yerekana ibisabwa bifatika byo gusuzuma: kugenzura uburyo abakozi babaza inyandiko zubuyobozi, bisaba itumanaho ryabakiriya ryeruye, gusuzuma ubundi buryo bwubaka, no kugenzura ibiciro bya serivisi mbere yo koherezwa. Inkomoko ntabwo itanga amakuru nyayo yoherejwe, ibisubizo byabakiriya, cyangwa impanuka z'umutekano, izo ngaruka rero ntizwi.

Ibijyanye nuyobora & ibibazo

Abakozi ba AIModeri ya AI YasobanuweAmahugurwa ya AIGerageza ibyo uzi - gerageza ikibazo cya AI kubuntuReba ijambo AI mumagambo yacuKurikiza icyerekezo cya AI cyo kurekura
Basanze ari ingirakamaro?