Chii chaitika
Vatsvakurudzi vakaunza computation-conditioned credit transport, nzira yekupa kusimbisa-kudzidza chikwereti kune zviratidzo zvega zvichienderana nekombuta yakaitwa nemuenzaniso wemutauro panguva yekupindura. Kuitwa kwavo, CompPO, inoshandisa kutarisisa kugadzira gedhi rekuchengetedza yega-yechiratidzo uye inoisanganisa nemutsoropodzi anoshandisa zvakare nyika dzakavanzika dzemutambi uye ruzivo rwekufambisa.
Preprint yakatumirwa kuarXiv musi wa21 Nyamavhuvhu inokurudzira nzira itsva yekupa chikwereti panguva yekusimbisa kudzidza kwemhando dzemitauro mikuru. Iro bepa rinopatsanura kiredhiti muzvikamu zvitatu: humbowo hwekuti kuburitswa kwakabudirira here, mutyairi wekutakura anoshandura humbowo uhu kuita mabhenefiti echiratidzo, uye iyo yekuvandudza geometry inoshandura izvo zvakanaka kuita shanduko. Vanyori vanopikisa kuti basa remazuva ano rakavandudza chikamu chekutanga uye chechitatu, nepo mitemo yekufambisa inowanzoshandiswa inoramba ichinyanya kuzvimiririra kune yemhando yekuvaka.
Iyo yakarongwa dhizaini, inodaidzwa kuti computation-conditioned kiredhiti yekufambisa, inoshandisa yakavharwa nhamba kubva mumaitiro emukati computation kuratidza kuti kukosha kwepasi kunotakurwa sei kuburikidza nekuburitswa. Iyo kongiri algorithm, CompPO, inoshandura kutarisisa kwekuzvarwa kuita gedhi rekuchengetedza rakasungwa rechiratidzo chega chega. Iro gedhi rinoshandiswa zvese kune imwe-nhanho bootstrapping uye kune nzira-inotsamira generalized-advantage trace inonzi Comp-GAE. Vanyori vanotaura kuti gedhi rinogara richidzikisa nzira kune yakagadziriswa-coefficient generalized advantage estimation, ichipa chinongedzo kune yakajairwa baseline.
CompPO inosanganisirawo mutsoropodzi wekutakura-akaenderana, kana TAC. Panzvimbo pekuwedzera Chechipiri Transformer yechiyero chimwe chete, TAC inoshandisazve nzvimbo dzakavanzwa dzemutambi uye ruzivo rwenzira. Iro bepa rinoti mubairo webasa uye chinangwa chePPO chakachekwa chinoramba chisina kuchinjika; iyo inonzi shanduko ndeyekuti chikwereti chinotakurwa sei uye kuti mutsoropodzi anowirirana sei nekutakura ikoko. Mukuedza kushandisa mhodzi shanu dzeQwen3-4B, vanyori vanoshuma 61.4% yekupedzisira-yakabata-yechokwadi, ine 95% yenguva yekuvimba ye60.8% kusvika 62.0%, zvichienzaniswa ne53.8%, ine nguva ye52.9% kusvika 54.7%, yetuned GRPO.
Mhedzisiro yekubvisa kwebepa inoratidza mhedzisiro yekusanganiswa kwezvikamu. Comp-GAE yakabatanidzwa nemutsoropodzi akajairwa yakasvika 55.2%, nepo TAC yakabatanidzwa negedhi rakagadziriswa yakasvika 56.4%; hapana chakaenderana nehurongwa hwakazara. Vanyori vanorondedzera kupindirana kwemaitiro e2.4 mapoinzi, ine 95% nguva yekuvimba ye1.9 kusvika 2.9 mapoinzi. Shuffle uye zvidzoreso zvenzvimbo zvakashumwa kutsigira trajectory-chaiyo kurongeka. CompPO yakanga yakagadzikana mu10 ye12 PPO-grid inomhanya, ichienzaniswa ne3 ye12 yekuenzanisa setup. Pakuongorora kwakaoma, vanyori vanotaura kuvandudzwa pamusoro peGRPO ye4.3 uye 3.9 yemakaro pass@1 macro point paQwen3-4B uye Llama-3.1-8B-Rayirira, zvichiteerana.
Nei zvichikosha
Mhedzisiro yacho inogadzirisa dambudziko rekusimbisa kudzidza kwemhando dzemitauro mikuru: kusarudza kuti ndezvipi zvikamu zvemhinduro refu zvinofanirwa kurumbidzwa kana mhosva yemhedzisiro yekupedzisira. Vanyori vanoshuma kuvandudzwa pamusoro peiyo GRPO yakagadziridzwa, asi humbowo hunobva kune preprint uye shoma seti yezviedzo, saka hunhu hwemaitiro uye mutengo wekushanda unoramba usina chokwadi.
Kusimbisa kudzidza kwemhando dzemitauro kunofanirwa kubatanidza mubairo wekupedzisira kune akawanda ega tokeni uye sarudzo dzepakati. Mhinduro inogona kuva nenhanho dzinobatsira uye dzisingabatsiri, asi nzira inotepfenyura zvimiro zvakafanana mumhinduro yese inogona kupa nhungamiro isina kusimba. Chirevo chepakati chebepa ndechekuti komputa yemuenzaniso pachayo inogona kupa chiratidzo chakanyanya chekusarudza kuti kiredhiti inofanira kuramba ichienderera sei kubva kune imwe tokeni kuenda kune inotevera.
Kana iyo yakashumwa mhedzisiro inobata munzvimbo dzakakura, dhizaini-inoziva kiredhiti yekufambisa inogona kuita yekusimbisa-yekudzidza zvigadziriso zvakanyanya kunangwa pasina kuda imwe tsananguro yemubairo kana chinangwa chegwaro. Izvo zvine basa nekuti shanduko yekupihwa kiredhiti inogona kuiswa mune iripo PPO-maitiro ekudzidzisa mapaipi. Kufananidzwa kwakataurwa neGRPO kunonyanya kukosha kune yazvino LLM yekusimbisa-kudzidza maitiro nekuti inogadzira nzira yakarongwa seshanduko kuchiratidzo chekudzidzisa kwete mhuri nyowani kana chigadzirwa chakatarisana nemushandisi.
Iwo akashumwa ablations anobatsira nekuti ivo vanoti mhedzisiro haitsanangurwe negedhi rakatorwa nehanya kana mutsoropodzi akashandiswa zvakare. Iyo yakazara nzira yakaitwa zvirinani pane ese ari maviri akasiyana mumhedzisiro yakapihwa, uye vanyori vanoti shuffle uye zvidzoro zvechinzvimbo zvinotsigira pfungwa yekuti trajectory-chaiyo kurongeka zvine basa. Kuenzanisa kwakadzikama kunonongedzerawo kune bhenefiti inogoneka: mashoma asina kugadzikana anomhanya anogona kuderedza akapambadzwa kuedza kudzidzisa, kunyangwe kwakabva kusingapi compute kana mutengo zviyero.
Humbowo hunofanirwa kuverengerwa semhedzisiro yekutsvaga yekutanga. Kunobva kwave kudhindwa kwearXiv, kwete chinyorwa chakaongororwa nevezera, uye chinyorwa hachitsanangure mabasa ari pasi, saizi yedhatabheti, ruzivo rwekutanga rwekuita, mabhajeti ekudzidzisa, kana maitiro ehuwandu kupfuura nguva dzakataurwa dzekuvimba. Izvo zvakare hazviratidze kana izvo zvakawana zvichiuya nekuwedzera ndangariro, latency, engineering kuoma, kana kunzwa kune yekutarisisa maitiro. Mabatiro everuzhinji saka zvinoenderana nekuti vaongorori vakazvimiririra vanogona kuburitsa mhedzisiro uye kana nzira yacho ichichinjisa kune akakura modhi uye akasiyana-siyana ekusimbisa-kudzidza zvinangwa.
Interactive Mechanism: Iyo Inonyatsoshanda
Ongorora ari pasi tekinoroji kuseri kwekusimudzira uku uchipindirana.
Which component of an AI application is the machine-learning model itself?
Zvekutarisa zvinotevera
Iwo akakosha anotevera bvunzo kudzokorora kwakazvimirira, modhi yakafara uye kufukidzwa kwebasa, uye kuenzanisa kunosanganisira mutengo wekudzidziswa, kushandiswa kwendangariro, uye nguva yekumhanya. Sosi yacho hairatidze kana CompPO inoshanda zvakavimbika kupfuura yakashumwa Qwen3-4B uye Llama-3.1-8B-Rayirira ongororo kana kuti chiratidzo chayo chekutarisisa chinoramba chichibatsira pamhando dzakasiyana dzezvivakwa.
Chekutanga chakanyanya kukoshesa kudzokorora kupfuura mhodzi shanu dzeQwen3-4B uye ongororo dzakaitwa nechando. Mapoka akazvimirira anofanirwa kuyedza nzira yacho pane mamwe masaizi emodhi, mabasa ekudzidzisa, zvimiro zvemubairo, uye mitauro-yemhando yekuvaka. Kwakabva mazita Qwen3-4B uye Llama-3.1-8B-Rayirira, asi haitauri kuti nzira yacho yakaongororwa mumhando dzakasiyana-siyana dzemhando here kana kuti chiratidzo chekutarisisa chinoita zvakafanana muzvivakwa zvine nzira dzakasiyana kana kutarisisa magadzirirwo.
Vatsvagiri vanofanirawo kuyera iyo yakazara kudzidziswa tradeoff. Iro bepa rinoti TAC inoshandisa zvakare mutambi akavanzika nyika uye routing ruzivo pasina yechipiri saizi-mwero Transformer, asi sosi haiverenge mashandisirwo endangariro, nguva yekudzidziswa kwewachi-wachi, zvinodiwa zvehardware, kana compute yakazara. Zviyero izvozvo zvichaona kana chokwadi chakataurwa uye kugadzikana kunowanikwa kunoshanda kumasangano anotoshandisa mapaipi anodhura ekusimbisa-kudzidza.
Rimwe basa rinofanira kuongorora kuti rakasimba sei gedhi rekuchengetedza rinobva kutarisisa. Iyo yakashumwa shuffle uye zvigaro zvinodzora zvinotsigira trajectory-chaiyo kurongeka mukati mezviedzo, asi kwakabva haaratidze kuti gedhi rinopindura sei kumamiriro akareba, maitiro asina kujairika ekufunga, mibairo mishoma kana ine ruzha, kana shanduko mukukurudzira uye sampling. Izvo hazvizivikanwe zvakare kana kutarisisa kuri proxy yakavimbika yekukosha kwekombuta kana kungori chiratidzo chinobatsira kune mamwe mamodheru uye mabasa akaedzwa.
Pakupedzisira, chimiro cheiyo nzira se preprint inokosha. Sosi yacho haitauri yakazvimiririra yekuongorora, kutumirwa kwekugadzira, kuwanikwa kwekodhi, kana ongororo yevezera. Izvo zvakasiiwa hazvikonese zvakawanikwa, asi zvinosiya mibvunzo yakakosha isina kupindurwa nezve kuberekana uye kuzere. Mhosva yakasimba yaizoda kuburitswa kweruzivo rwekuita, kufananidzwa-mutengo wekutanga, mibairo kune mamwe mavakirwo, uye humbowo hwekuti kuvandudzwa kunoenderera kunze kwekumisikidzwa kwevanyori.