Kwenzekeni
Abacwaningi bethule i-computing-conditioned credit transport, indlela yokunikeza ikhredithi yokufunda yokuqinisa amathokheni ngamunye ngokuya ngokubala okwenziwe imodeli yolimi ngesikhathi sokuphendula. Ukuqaliswa kwabo, i-CompPO, kusebenzisa ukugxilisa ukunakwa ukuze kudale isango lokugcinwa kwethokheni ngayinye futhi kulihlanganisa nomgxeki ophinde asebenzise izifunda ezifihliwe zomlingisi nolwazi lomzila.
Ukuphrinta kusengaphambili okuthunyelwe ku-arXiv ngo-Agasti 21 kuhlongoza indlela entsha yokunikeza isikweletu ngesikhathi sokufunda okuqinisiwe kwamamodeli ezilimi ezinkulu. Iphepha lihlukanisa isabelo sesikweletu sibe izingxenye ezintathu: ubufakazi mayelana nokuthi ukukhishwa kuphumelele yini, isisebenzisi sezokuthutha esiguqula lobo bufakazi bubenze izinzuzo zeleveli yamathokheni, kanye nejiyomethri yokubuyekeza eguqula lezo zinzuzo zibe izinguquko zenqubomgomo. Ababhali bathi umsebenzi wakamuva uthuthukise ingxenye yokuqala neyesithathu, kuyilapho imithetho yokuthutha esetshenziswa ngokuvamile ihlala izimele kakhulu ekwakhiweni kwemodeli.
Uhlaka oluhlongozwayo, olubizwa nge-computation-conditioned credit transport, lusebenzisa izibalo ezihlukanisiwe kusukela ekubalweni kwangaphakathi kwenqubomgomo yokuziphatha ukuze kubekwe ipharamitha ukuthi inani eliphansi lomfula lithuthwa kanjani ngokukhishwa. I-algorithm yekhonkrithi, i-CompPO, iguqula ukugxila komdabu kube isango lokugcinwa eliboshiwe lethokheni ngayinye. Lelo sango lisetshenziselwa kokubili isinyathelo esisodwa sokuqalisa nokulandela umkhondo wenzuzo evamile oncike endleleni ebizwa nge-Comp-GAE. Ababhali basho ukuthi isango elingaguquki linciphisa indlela eya esilinganisweni senzuzo evamile esine-coefficient engashintshi, sinikeza isixhumanisi kusisekelo esijwayelekile.
I-CompPO iphinde ihlanganise nomgxeki oqondaniswe nezokuthutha, noma i-TAC. Esikhundleni sokwengeza i-Transformer yesibili yesikali esifanayo, i-TAC iphinda isebenzise izimo ezifihliwe zomlingisi nolwazi lomzila. Iphepha lithi umvuzo womsebenzi kanye nenhloso yenqubomgomo ye-PPO enqanyuliwe kuhlala kungashintshile; uguquko olufunwayo luwukuthi isikweletu sithuthwa kanjani nokuthi umgxeki uhambisana kanjani nalokho kuthutha. Ocwaningweni olusebenzisa imbewu ye-Qwen3-4B emihlanu, ababhali babika ukunemba kokugcina okungu-61.4%, ngesikhawu sokuzithemba esingu-95% sika-60.8% kuya ku-62.0%, uma kuqhathaniswa no-53.8%, nesikhawu esingu-52.9% kuya ku-54.7%, se-GRPO eshuniwe.
Imiphumela yokukhishwa kwephepha ihlanganisa umphumela ekuhlanganisweni kwezingxenye. I-Comp-GAE ebhangqwe nomgxeki ojwayelekile ifinyelele ku-55.2%, kuyilapho i-TAC ebhangqwe nesango elingashintshi ifinyelele ku-56.4%; akukho okufana nesistimu egcwele. Ababhali babika umthelela wokusebenzisana wamaphoyinti angu-2.4, ngesikhawu sokuzithemba esingu-95% samaphuzu angu-1.9 kuya kwangu-2.9. Ukushova nezilawuli zendawo kubikwe ukuthi zisekela ukuqondanisa kwe-trajectory ethize. I-CompPO ibizinzile ku-10 yokugijima kwe-PPO-grid engu-12, uma kuqhathaniswa noku-3 kokungu-12 ekusethweni kokuqhathanisa. Ekuhlaziyeni okufriziwe, ababhali babika intuthuko ngaphezu kwe-GRPO engu-4.3 kanye ne-3.9 ehahayo pass@1 amaphuzu ama-macro ku-Qwen3-4B kanye ne-Llama-3.1-8B-Instruct, ngokulandelanayo.
Imininingwane yomthombo: arxiv.org ↗
Kungani kubalulekile
Umphumela ubhekana nengqinamba engokoqobo ekuqiniseni ukufunda kumamodeli amakhulu olimi: ukunquma ukuthi yiziphi izingxenye zempendulo ende ezifanelwe ukunconywa noma ukusolwa ngomphumela wokugcina. Ababhali babika intuthuko ku-GRPO eshuniwe, kodwa ubufakazi buvela ekuprintweni kwangaphambili kanye nesethi elinganiselwe yokuhlola, ngakho-ke ukujwayelekile kwendlela kanye nezindleko zokusebenza kuhlala kungaqinisekile.
Ukufunda okuqiniswayo kwamamodeli olimi kufanele kuxhume umvuzo wokugcina kumathokheni amaningi ngamunye kanye nezinqumo eziphakathi nendawo. Impendulo ingaqukatha izinyathelo eziwusizo nezingelona usizo, kodwa indlela esakaza izibalo zomphumela ofanayo kuyo yonke impendulo ingase inikeze isiqondiso esibuthakathaka. Isimangalo esiyinhloko sephepha ukuthi ukubala kwemodeli ngokwayo kunganikeza isignali ecacile yokunquma ukuthi isikweletu kufanele siqhubeke kangakanani ukusuka kwelinye ithokheni ukuya kwelinye.
Uma umthelela obikiwe ubambele kuzilungiselelo ezibanzi, ezokuthutha zekhredithi eziqaphela izakhiwo zingenza izibuyekezo zokuqinisa ukufunda ziqondiswe kakhulu ngaphandle kokudinga incazelo ehlukile yomvuzo noma inhloso yenqubomgomo. Lokho kubalulekile ngoba izinguquko kumsebenzi ozokwenziwa wekhredithi zingase zifakwe kumapayipi okuqeqesha akhona esitayela se-PPO. Ukuqhathanisa okubikiwe ne-GRPO kubaluleke kakhulu kumkhuba wamanje wokufunda wokuqinisa we-LLM ngoba uzimele indlela ehlongozwayo njengoshintsho kusiginali yokuqeqesha esikhundleni somndeni oyimodeli entsha noma umkhiqizo obheke umsebenzisi.
Ukukhishwa okubikiwe kuyasiza ngoba kuphakamisa ukuthi umphumela awuchazwa yisango elisuselwe ukunakwa noma umgxeki ophinde wasetshenziswa yedwa. Indlela egcwele yenze kangcono kunokwahluka kokubili ngokwengxenye emiphumeleni enikeziwe, futhi ababhali bathi ukushova nezilawuli zendawo zisekela umbono wokuthi ukuqondanisa okuqondene ne-trajectory kuyabalulekile. Ukuqhathaniswa kokuqina futhi kukhomba kunzuzo engaba khona engokoqobo: ukugijima okumbalwa okungazinzile kunganciphisa imizamo yokuqeqeshwa emoshiwe, nakuba umthombo unganikezi izilinganiso zokubala noma izindleko.
Ubufakazi kusafanele bufundwe njengomphumela wocwaningo lwangaphambi kwesikhathi. Umthombo ukuphrinta kusengaphambili kwe-arXiv, hhayi ukushicilelwa okubuyekezwe ngontanga, futhi i-abstract ayichazi imisebenzi eyisisekelo, osayizi bedathasethi, imininingwane eyisisekelo yokuqalisa, ibhajethi yokuqeqesha, noma izinqubo zezibalo ezingaphezu kwezikhawu zokuzithemba ezibikiwe. Futhi ayiqinisekisi ukuthi izinzuzo ziza nenkumbulo eyengeziwe, ukubambezeleka, ubunkimbinkimbi bobunjiniyela, noma ukuzwela kumaphethini wokunaka. Ngakho-ke umthelela womphakathi wendlela uncike ekutheni abacwaningi abazimele bangakwazi yini ukukhiqiza kabusha imiphumela nokuthi indlela idlulisela kumamodeli amakhulu kanye nezinjongo ezihlukene zokufunda ukuqinisa.
I-Interactive Mechanism: Indlela Esebenza Ngayo Ngempela
Hlola ubuchwepheshe obuyisisekelo ngemuva kwalokhu kuthuthukiswa ngokuhlanganyela.
Which component of an AI application is the machine-learning model itself?
Ongakubuka ngokulandelayo
Ukuhlola okulandelayo okubalulekile ukuphindaphinda okuzimele, imodeli ebanzi nokufakwa komsebenzi, nokuqhathanisa okufaka izindleko zokuqeqesha, ukusetshenziswa kwenkumbulo, nesikhathi sokusebenza. Umthombo awuqinisekisi ukuthi i-CompPO isebenza ngokuthembekile yini ngale kokuhlolwa kwe-Qwen3-4B ne-Llama-3.1-8B-Yala okubikiwe noma ukuthi isignali yayo esekelwe ekunakekelweni ihlala iwusizo kuwo wonke amamodeli ezakhiwo.
Okubalulekile kokuqala ukuphindaphinda ngale kwezimbewu ezinhlanu ze-Qwen3-4B kanye nokuhlola okubikiwe okufriziwe. Amaqembu azimele kufanele ahlole indlela ngosayizi bemodeli abengeziwe, imisebenzi yokuqeqesha, izakhiwo zokuklomelisa, nezakhiwo eziyimodeli yolimi. Umthombo uqamba i-Qwen3-4B kanye ne-Llama-3.1-8B-Instruct, kodwa ayisho ukuthi indlela iye yahlolwa kulo lonke uhla olubanzi lwamamodeli noma ukuthi isignali yokunaka iziphatha ngendlela efanayo ekwakhiweni kwezakhiwo ezinomzila ohlukile noma imiklamo yokunaka.
Abacwaningi kufanele futhi balinganise ukuhweba okugcwele kokuqeqeshwa. Leli phepha lithi i-TAC iphinda isebenzise izifunda ezifihliwe zomlingisi kanye nolwazi lomzila ngaphandle kwe-Transformer yesibili yesikali esifanayo, kodwa umthombo awubali ukusetshenziswa kwenkumbulo, isikhathi sokuqeqeshwa kwewashi lodonga, izidingo zehadiwe, noma ikhompuyutha ephelele. Lezo zilinganiso zizonquma ukuthi ukunemba okubikiwe kanye nezinzuzo zokuzinza ziyasebenza yini ezinhlanganweni esezivele zisebenzisa amapayipi okufunda okuqinisa abizayo.
Umsebenzi owengeziwe kufanele uhlole ukuthi liqine kangakanani isango lokugcina elisuselwa ekunakekelweni. Ukushova okubikiwe nezilawuli zendawo zisekela ukuqondanisa okuqondile kwe-trajectory ngaphakathi kokuhlolwa, kodwa umthombo awubonisi ukuthi isango lisabela kanjani kumongo omude, ukulandela ngomcabango okungajwayelekile, imiklomelo eyingcosana noma enomsindo, noma izinguquko ekwazisweni nasekuthathweni kwesampula. Akwaziwa futhi ukuthi ukugxilisa ukunaka kuwummeleli othembekile wokubaluleka kwekhompyutha noma isignali nje ewusizo yamamodeli athile nemisebenzi ehloliwe.
Okokugcina, isimo sendlela njenge-preprint sibalulekile. Umthombo awukubiki ukuqinisekiswa okuzimele, ukusetshenziswa kokukhiqiza, ukutholakala kwekhodi, noma imiphumela yokubuyekezwa kontanga. Lokho okweqiwe akukwenzi kube yize okutholakele, kodwa kushiya imibuzo ebalulekile ingaphendulwanga mayelana nokuphindaphindeka kanye nokuhlanganisa. Icala eliqinile lingadinga imininingwane ekhishiwe yokuqaliswa, isisekelo sezindleko ezifanisiwe, imiphumela kuzo zonke izakhiwo ezengeziwe, kanye nobufakazi bokuthi ukuthuthukiswa kuyaqhubeka ngaphandle kokuhlelwa kokuhlola kwababhali.