Kwenzekeni
Abacwaningi uMichael Wu kanye no-Arquimedes Canedo bahlongoza "izinkontileka ezingavumelekile," ungqimba lwephrothokholi lwabasebenzeli be-AI olugcina inqolobane yeziphakamiso zokutholwa kumaphutha e-API kuzo zonke iziqephu. Izinkontileka zinamathisela izitembu zenguqulo kanye namacebo okugcineka kunqolobane esiphakamisweni ngasinye ukuze amaklayenti akwazi ukususa okufakiwe okukudala ngemva kokukhukhuleka kwedatha yohlangothi lweseva kuyilapho elondoloza okufakiwe okuhlala kusebenza.
Iphepha likhuluma nabasebenzeli be-AI abagcina iziphakamiso zokutakula ngemva kokuhlangabezana namaphutha e-API. Ukukhathazeka kwayo okuyinhloko ukuthi isiphakamiso esisebenze esiqeshini esisodwa singase sibe nephutha ngemva koshintsho lwedatha yohlangothi lweseva. Ukuthola kabusha ukulungiswa kuzo zonke iziqephu kungagwema leyo modi yokwehluleka, kodwa ababhali bathi kubuyisela ukonga okuvela kunqolobane. Isixazululo sabo esihlongozwayo, esibizwa ngokuthi izinkontileka zokungavumeleki, sengeza imethadatha yephrothokholi esiphakamisweni ngasinye sokutakula esikhundleni sokuphatha inqolobane njengebhulokhi elingahlukanisiwe.
Inkontileka inamathisela izitembu zenguqulo kanye namacebiso okugcinekayo eziphakamisweni. Iklayenti libe selikhipha okufakiwe okuhlobene nedatha eshintshiwe kuyilapho ligcina okufakiwe okusasebenza. Ababhali bahlukanisa ukonga okuwumphumela kube yiqiniso—ingxenyana yeziphakamiso ezigcinwe kunqolobane ezihlala zilungile ngemva komcimbi we-drift—kanye nokuthobelana—ingxenyana umhleli eyisebenzisa ngendlela efanele emzamweni wayo wokuqala. Iphepha lithi ukufaneleka kunqunywa iphrothokholi futhi kuzimele kumthengisi, kuyilapho ukuthobela kuncike kumodeli yomhleli.
Ukuhlola kuhlanganisa amamodeli ayisikhombisa, izindlela ezintathu zokusebenzela, izizinda ezimbili kanye neziqephu ezingaba ngu-9,400. Imibiko efingqiwe yokuthi ukungavumeleki kwezinga lomugqa kwenyuse ukuthobela umthetho ngamaphesenti aphakathi kuka-0 nangu-66.7 wamaphoyinti kuwo wonke amamodeli; amamodeli amathathu abone ukuzuza phakathi kwamaphuzu angama-55.6 nama-66.7. Amamodeli amane kwayisikhombisa athole u-29% kuya ku-33% wezindleko zamathokheni ayisisekelo. Iphepha liphinde libike ukunemba okuphelele kokukhishwa, 1.00, kubumbudumbudu bomugqa ngaphansi kwe-oracle yeleveli echazwe kuSigaba 4.1.
Imiphumela yahlukahluka kakhulu ngamamodeli. Ababhali babika ukuthobelana kokuzama kokuqala okungu-100% kwe-Claude Haiku 4.5 ngamabhayithi ocingo afanayo, uma kuqhathaniswa no-11% noma ngaphansi kwe-Claude Sonnet 5. Babeka ukuziphatha kwe-Sonnet ku-conservatism ye-input-schema: ukwenqaba ukulungisa okwengeza izinkambu ezingekho ku-original. Ngokuphambene, ukungavumeleki kweleveli yethebula kucekele phansi okufakiwe okubekwe ndawonye futhi kwehlise izinga lokuzama kokuqala kwangemva kokukhukhuleka ukuya ku-0% kumamodeli amahlanu kwayisikhombisa. Inkontileka yengeze u-15% emalini ekhokhelwa impendulo, futhi ababhali babika ukuthi akukho ukwehluleka kwenkontileka kukho konke ukuhlola.
Imininingwane yomthombo: arxiv.org ↗
Kungani kubalulekile
Iphepha lifaka inkumbulo yomenzeli ephikelelayo njengenkinga yokuthembeka nokusebenza kahle: ukulungisa kabusha ukubala kabusha ngaso sonke isikhathi kugwema iseluleko esidala kodwa kudela ithokheni nokonga kwemodeli okungahlinzekwa yinqolobane. Imiphumela yayo ebikiwe iphakamisa ukuthi ukwenza inkumbulo engavumelekile ezingeni lomugqa kungagcina okufakiwe kwenqolobane okuwusizo, nakuba okutholiwe kuvela ekuhlolweni kokuphrinta okukodwa futhi akusunguli ukusebenza kokukhiqiza.
Indaba engokoqobo ayikona nje ukuthi i-ejenti ingakwazi ukukhumbula. Ukuthi imemori ihlala iphephile yini ukusetshenziswa uma isevisi yangaphandle ishintsha. Isiphakamiso sokutakula esifakwe kunqolobane singanciphisa ukucabanga okuphindaphindiwe, ukusetshenziswa kwamathokheni namakholi emodeli, kodwa isinqamuleli esifanayo singaba ukwehluleka buthule uma i-API ezungezile noma idatha ikhukhulekile. Iphrothokholi ehlongozwayo yenza ulwazi olusha lube yingxenye yokusebenzelana phakathi kwesevisi neklayenti.
Umehluko wephepha phakathi kokufaneleka nokuthobela uwusizo ngoba uhlukanisa imithombo emibili yokwehluleka. Iphrothokholi ingakwazi ukukhomba ukuthi yikuphi okufakiwe okufakwe kunqolobane okufanele kulahlwe, kodwa umenzeli angase ahluleke ukusebenzisa isiphakamiso esivumelekile. Umehluko obikiwe phakathi kwe-Haiku 4.5 ne-Sonnet 5 ubonisa ukuthi ukuklanywa kwephrothokholi kukodwa kungase kunganqumi ukuthi umhleli uyazuza yini kumemori egciniwe. Ukuziphatha kwemodeli kuhlala kuyisithiyo esihlukile sokusebenza.
Imiphumela yezinga lerowu ebikiwe iphakamisa isimiso sokuklama esingaba khona samasistimu agcina iziphakamiso eziningi zokutakula ndawonye: vala okufakiwe okuthintekile kuphela uma umthetho olandelwayo ungakukhomba. Ukuqhathaniswa kweleveli yethebula kubalulekile phakathi kokuhlolwa kwephepha ngoba ukungavunyelwa okubanzi kususe okufakiwe okungahlobene futhi kwakhiqiza amanani okuzama kokuqala angemva kokukhukhuleka kumamodeli amaningi ahloliwe. Lokho kutholwa, uma kuphindwa, kungabaluleka ngezindleko nokuthembeka kwama-ejenti asebenza isikhathi eside.
Isimangalo sokusebenza kahle sinqunyelwe. Ababhali babika ukubuyiswa kwe-29% kuya ku-33% yezindleko zethokheni eyisisekelo kumamodeli amane, hhayi kuzo zonke eziyisikhombisa, futhi iphrothokholi yengeza i-15% ekuphenduleni okukhokhelwayo. Umthombo awusho ukuthi lezo zindleko zihumusheka kanjani ku-latency, umkhawulokudonsa, izindleko zengqalasizinda noma ukwethembeka okubonakalayo komsebenzisi ekukhiqizeni. Futhi i-abstract ayisho ukuthi izizinda ezihloliwe noma izindlela ezisetshenziswayo zimele ububanzi bama-API asetshenziswa ama-ejenti asetshenzisiwe.
Lokhu ukuphrinta kusengaphambili kunomphumela wokukhiqiza oqinisekisiwe ngokuzimele. Umthombo ukhomba usayizi wokuhlola nemiphumela yesihloko kodwa awunikezi ubunikazi bamamodeli ayisikhombisa kwayisikhombisa, amagama ezizinda ezimbili, izimo ze-drift eziqondile, noma izilinganiso zokungaqiniseki. Ukunemba okuphelele kokukhipha kuboshwe ngokusobala ku-oracle yezinga lomugqa, ngakho akufanele kufundwe njengobufakazi bokuthi amaklayenti asetshenzisiwe azohlala azi ukuthi yimiphi imigqa edala.
I-Interactive Mechanism: Indlela Esebenza Ngayo Ngempela
Hlola ubuchwepheshe obuyisisekelo ngemuva kwalokhu kuthuthukiswa ngokuhlanganyela.
An agent must create a draft calendar event for Tuesday at 2 p.m. Which evidence would establish the requested result?
Ongakubuka ngokulandelayo
Imibuzo eyinhloko evulekile ukuthi ingabe iphrothokholi isebenza nabahleli abaningi, izinsizakalo, izizinda kanye namaphethini omhlaba wangempela wokukhukhuleka, kanye nokuthi ingabe u-15% wayo wenkokhelo yokuphendula wamukelekile ezinhlelweni ezisetshenzisiwe. Ukucutshungulwa okwengeziwe kufanele futhi kuhlole ukuhlolwa kokuxoshwa okusekelwe ephepheni kwe-oracle, amamodeli angaziwa phakathi kwayisikhombisa ahloliwe, kanye nezimbangela zegebe elikhulu lokuthobelana phakathi kwamamodeli aqanjwe Claude.
Ukuphindaphinda kufanele kuhlole ukuthi ingabe ukufaneleka kwesitembu senguqulo kuhlala kuyisinqumo uma amasevisi ashintsha ama-schema, ama-semantics noma idatha ngamanani ahlukene. Iphepha libika imiphumela efanayo yokufaneleka kuyo yonke imodeli nezindlela zokusebenza kanye nokwehluleka kwenkontileka okuyiziro, kodwa umthombo awuchazi ububanzi bemicimbi ye-drift ngemininingwane eyanele ukuze unqume ukuthi lowo mphumela usebenza kabanzi kangakanani.
Ukuthobela amamodeli ohlangothini kudinga uphenyo oluhlukene. Igebe elibikiwe eliphakathi kokungu-100% kokuthotshelwa kokuzama kokuqala ku-Claude Haiku 4.5 kanye no-11% noma ngaphansi ku-Claude Sonnet 5 liphakamisa ukuthi ama-ejenti angahumusha ukukhokhelwa kwephrothokholi efanayo ngendlela ehlukile. Ukuhlola okuzayo kufanele kucacise ukuthi ingabe inkinga iqondene ngqo namamodeli aqanjiwe, ukwazisa noma ukwakheka kwe-schema, noma ukuthambekela okubanzi phakathi kwabahleli bokwenqaba ukulungisa okungajwayelekile kwesakhiwo.
Ukushintshaniswa kwezindleko kufanele kukalwe ngaphezu kwamathokheni. Ukukhushulwa kokukhokha ngo-15% kungase kube kuncane noma kube okubalulekile kuye ngobude besiqephu, izimo zenethiwekhi kanye nokuthi iziphakamiso ezifakwe kunqolobane ziphinde zisetshenziswe kangaki. Umthombo ubika ukutholwa kwezindleko zamathokheni kumamodeli amane kodwa awuhlinzeki ngemiphumela yokubambezeleka, umkhawulokudonsa, imali noma isilinganiso sokwehluleka, ngakho lezo zihlala zingaziwa eziphusile kubaqalisi.
I-oracle yezinga lomugqa ingelinye iphuzu okufanele libhekisiswe. Ukunemba okuphelele ngaphansi kwe-oracle kubonisa ukuziphatha kwembudumbudu ehlongozwayo lapho umugqa othintekile waziwa, kodwa akuqinisekisi ukuthi iklayenti langempela lingakwazi ukuhlonza umugqa olungile kumasignali wesevisi evamile. Ubufakazi mayelana nokutholwa okuzenzakalelayo, okungalungile okungamanga kanye nemibono engamanga bunganquma ukuthi ingakanani inzuzo embikiwe esindayo ngaphandle kokusetha kokuhlola.
Okokugcina, umsebenzi kufanele uqhathaniswe nezinye izindlela eziqhubekayo zesimo se-ejenti kanye nokungasebenzi kwenkumbulo ngaphansi kwemisebenzi efanayo nezimo zokukhukhuleka. Umthombo usungula isiphakamiso sephrothokholi kanye nebhentshimakhi ebikiwe, kodwa awuqinisekisi ukutholakala kokuphakelwa, ukutholwa, ukuphindaphinda okuzimele noma ukuphakama kunezinye izindlela ezingashiwongo.