Kwenzekeni
Abacwaningi bashicilele i-MMPIBench, ibhentshimakhi ehlola ukuhlaselwa komjovo osheshayo we-multimodal kuzinhlaka ze-AI ze-ajenti. Ucwaningo luhlole izinhlaka eziyisithupha kanye namamodeli ayisisekelo amahlanu kuzo zonke iziteshi ezibukwayo nezomsindo, bathola ukuthi nakuba ukuhlasela okubukwayo kuvinjelwe kakhulu ngesikhathi sokuhlela, ukuhlaselwa komsindo kunezinga eliphezulu kakhulu lokuqeda lapho ingqalasizinda ibasekela.
Abacwaningi bethule i-MMPIBench, ibhentshimakhi ephindaphindekayo eklanyelwe ukukala umthelela wokuhlaselwa komjovo osheshayo we-multimodal kuzinhlaka ze-AI ze-agent. Lezi zinhlaka zivumela amamodeli olimi ukuthi ahlele, agcine inkumbulo, futhi ashaye amathuluzi asebenzisana namafayela omhlaba wangempela, ama-imeyili, namasevisi. Ibhentshimakhi iletha isethi engaguquki yokuhlasela ngezinkampani zenethiwekhi eziyisithupha ezibukwayo, okuhlanganisa umbhalo we-OCR, izimbondela, imethadatha ye-EXIF, amakhodi e-QR, ukuxhumana okungelona iqiniso, nama-hybrids, alandelela ukuthi imiyalelo ejovwe isuka kude kangakanani ukusuka ekuboneni ngokuhlela ukuya ekusebenzeni kwamathuluzi.
Ucwaningo lwenze ama-run angu-720 ahlanganisa izinhlaka eziyisithupha, amamodeli esisekelo amahlanu, abathwali abayisithupha, kanye nezinjongo zokuhlasela ezine. Imiphumela ibonise ukuthi ukuhlasela kuqede cishe u-1% wama-run kodwa kuzanywe ngo-12.8%. Igebe phakathi kokuhlaselwa okuzanyiwe nokuqediwe livalwe cishe ngokuphelele esinyathelweni sokuhlela, lapho imodeli ifunde umyalo ojovwe futhi yenqaba ukwenza. Imodeli ethile yesisekelo esetshenzisiwe yayibaluleke kakhulu kunohlaka lokuthi ingabe umyalelo wenziwa yini; imodeli eyodwa ayizange izame ukuhlasela futhi yabona umjovo kuma-run angu-59.7%, kanti amanye amabili azame ukuhlasela kuma-runs angu-23.6%.
Abacwaningi banwebe ibhentshimakhi kumsindo, okuwukuphela kwesiteshi sombono esingahluziwe esamukelwa amamodeli amanje asemngceleni. Amabili kuphela amamodeli amahlanu angenise umsindo, futhi ezintathu kuphela izinhlaka eziyisithupha eziwulethile. Kodwa-ke, lapho isignali ifika khona, ukuhlasela kuqedwe ku-49% wamaseli, akhuphukela ku-75% ngemodeli eyodwa ethile. Lokhu kubonisa ukuthi nakuba iziteshi ezibukwayo zivikelwa kakhulu ngokuhlela okunengqondo, iziteshi zomsindo zincane kodwa azivikeleki kancane, okuholela emazingeni aphezulu empumelelo ngeziyalezo ezinonya.
Imininingwane yomthombo: arxiv.org ↗
Kungani kubalulekile
Lolu cwaningo lunikeza ubufakazi obuphathekayo, obuphindaphindekayo bokuthi amasistimu e-AI e-agent, angafinyelela amafayela nezinsizakalo zangempela, ahlala esengozini yokujova ngokungaqondile ngomjovo osheshayo ngeziteshi ezingezona ezombhalo. Okutholakele kugqamisa igebe elibalulekile lokuvikela: kuyilapho imijovo ebonakalayo ivamise ukuncishiswa ngokucabanga okuyimodeli, iziteshi zomsindo zivikeleke kancane futhi zingaholela ezingcingweni zamathuluzi ezinonya eziphumelelayo. Lokhu kugcizelela isidingo sokufakwa kwenhlanzeko kokufakwayo okuqinile kanye nokuqeqeshwa kokuphepha kwezindlela eziningi ekusetshenzisweni kwe-ejenti.
Ucwaningo lubonisa ukuthi ukubika kuphela amazinga okuphothula ukuhlasela kwehlisa ukuchayeka kwangempela kwezokuvikela kwamasistimu e-AI e-agent. Izinga eliphezulu lokuzama ukuhlasela (12.8%) uma kuqhathaniswa nokuqediwe (1%) liphakamisa ukuthi amamodeli amanje avame ukubona inhloso enonya kodwa lokhu kuvikela akufani kuwo wonke amamodeli noma izindlela.
Ukuthola ukuthi ukuhlasela okusekelwe kumsindo kunezinga lokuqedwa elingu-49% ezindaweni ezisekelwayo kuthinta ikakhulukazi ngoba umsindo uyisiteshi sokufakwayo esingajwayelekile sokuthunyelwa okuningi kwe-ejenti, okusho ukuthi ungathola ukucutshungulwa kokuphepha okuncane. Lokhu kudala indawo engaba yimpumputhe lapho abahlaseli bengase badlule izivikelo ezibukwayo ngokusebenzisa okokufaka komsindo ukuze bakhohlise abenzeli ukuba benze amakholi amathuluzi ayingozi.
Ukwehluka phakathi kwamamodeli, ngemodeli eyodwa ebona imijovo cishe ku-60% wokugijima kanti amanye azama ukuhlasela ngaphezu kuka-20%, kugqamisa ukuthi ukukhethwa kwemodeli kuyisici esibalulekile ekuvikelekeni kwe-AI ye-ajenti. Izinhlangano azikwazi ukuthembela kuphela ekuvikelweni kwezinga lohlaka futhi kufanele zicabangele izici ezithile zokuphepha zamamodeli esisekelo ayisisekelo.
I-Interactive Mechanism: Indlela Esebenza Ngayo Ngempela
Hlola ubuchwepheshe obuyisisekelo ngemuva kwalokhu kuthuthukiswa ngokuhlanganyela.
An agent must create a draft calendar event for Tuesday at 2 p.m. Which evidence would establish the requested result?
Ongakubuka ngokulandelayo
Onjiniyela namabhizinisi asebenzisa i-AI yomenzeli kufanele aqaphe izibuyekezo ze-MMPIBench namapeshi okuphepha ahlobene. Imboni ingase ibone ukugxila okuthuthukisiwe ekuhlanzweni komsindo okufakwayo kanye nezimvume eziqinile zamakholi wamathuluzi ukuphendula lokhu okutholakele.
Gada ukuze uthole izibuyekezo ze-MMPIBench kanye nocwaningo olulandelayo olungase luhlole imigudu yokuqonda eyengeziwe noma ama-vector okuhlasela ayinkimbinkimbi. Ukukhiqizwa kabusha kwebhentshimakhi kuvumela ukuqinisekiswa komphakathi okuqhubekayo nokunwetshwa.
Buka izimpendulo zemboni ezivela kubathuthukisi abakhulu bohlaka lwe-AI nabahlinzeki bemodeli yesisekelo, abangase bakhulule amapeshi okuvikela noma ukuqeqeshwa okuthuthukisiwe kokuphepha ukuze kubhekwane nobuthakathaka obuthile obukhonjwe eziteshini ezilalelwayo nezibonwayo.
Qaphela ukuthi amabhizinisi asebenzisa i-AI ye-ejenti azilungisa kanjani izimiso zawo zokuphepha, okungenzeka asebenzise ukuhlunga okokufaka okuqinile kwedatha okungeyona eyombhalo kanye nezilawuli zemvume ye-granular yamakholi amathuluzi ukuze kwehliswe izingozi eziqokonyiswa ucwaningo.