Ukuthuthukisa Okuncamelayo Okuqondile
I-Direct Preference Optimization (DPO) iyindlela yokuvumelanisa amamodeli olimi nokuthandwayo komuntu ngaphandle kokuqeqesha imodeli yomvuzo ehlukile noma ukuqhuba ukufunda okuqiniswayo.
Uhlolojikelele
It collapses a complex multi-stage pipeline into a single, stable training loss.
I-Deep Dive
I-DPO, eyethulwe u-Rafailov kanye nozakwabo e-Stanford ngo-2023, icabanga kabusha ukuthi sifundisa kanjani imodeli lokho abantu abakuthandayo. Indlela yendabuko (RLHF) iqeqesha imodeli yomvuzo ekuqhathaniseni abantu, bese isebenzisa ukufunda okuqiniswayo ukuze kwandiswe lowo mvuzo. Ukuqonda okubalulekile kwe-DPO kungokwezibalo: inqubomgomo ephelele ngaphansi kwaleyo nhloso ye-RLHF inobudlelwano obungajwayelekile nomklomelo, ukuze ukwazi ukuhlela kabusha izibalo futhi ulungiselele imodeli yolimi ngokuqondile kumapheya okuncamelayo. Ukunikeza ukwaziswa, impendulo 'ekhethiwe' (ekhethwayo), kanye nempendulo 'enqatshiwe', futhi ukulahlekelwa okulula kwesitayela sokuhlukanisa kugudluza imodeli ukuze kwenziwe impendulo ekhethiwe ibe maningi kakhulu. Ayikho imodeli yomvuzo, ayikho i-loop yesampula, akukho ukugebenga komvuzo. Kulula kakhulu futhi kuzinzile ukugijima.
I-Technical Insight
I-DPO isebenzisa ukulahleka kwe-cross-entropy kanambambili kunamapheya athandwayo. Yenyusa isilinganiso samathuba elogi yempendulo ekhethiwe ngokuhlobene naleyo enqatshiwe, ngayinye ikalwa ngemodeli yesithenjwa efriziwe (imvamisa indawo yokuqala egadiwe-elungiswe kahle). I-beta yepharamitha yezinga lokushisa ilawula ukuthi inqubomgomo ingase isuke kude kangakanani kuleso sithenjwa, iphoqelela ngokusobala umkhawulo we-KL osetshenziswa i-RLHF ngokusobala. Umvuzo awukaze ubonakale; kusobala emathubeni okungena enqubomgomo.
I-Strategic Impact
Isivinini nesikali
Ukugeleza komsebenzi wolimi kungahamba ngokushesha ngaphandle kokudela ukuvumelana.
Finyelela futhi ufinyelele
Yandisa ukufinyelela kuzo zonke izilimi nezitayela zokuxhumana.
Izinqumo ezicacile
Amaqembu angachitha isikhathi esiningi ekwahluleleni kuyilapho i-automation isingatha impinda.
Ikusasa Lokuthuthukisa Okuthandwayo Okuqondile
I-DPO isiphenduke indlela yokuqondanisa ezenzakalelayo ngenxa yokuthi ishibhile futhi iyakhiqizeka, futhi yazala umndeni wezinhlobonhlobo: I-IPO ilungisa ukugcwala ngokweqile kokuncanyelwayo okucishe kunqunywe, i-KTO ifunda kumalebula awodwa angcono noma amabi esikhundleni samapheya, futhi i-ORPO igoqa okuthandwayo kokufunda ukuze kulungiswe kahle ngaphandle kwemodeli yereferensi. Lindela umsebenzi oqhubekayo wokuhlanganisa i-DPO nedatha yenqubomgomo kanye nobude/ukwehliswa kwekhwalithi, unciphise igebe elisele nge-RLHF eku-inthanethi egcwele.
Ukuqaliswa Komhlaba Wangempela
Ukushuna kahle amamodeli ezingxoxo anesisindo esivulekile afana ne-Zephyr kanye nokunye okuphuma ku-Llama ne-Mistral okuningi, okuhambisana ne-DPO kumasethi edatha athandwayo.
Ukunciphisa imiphumela eyingozi noma engelona usizo usebenzisa amapheya lapho impendulo ephephile, ewusizo 'ikhethwa' phezu kwenkinga
Ukufundisa umsizi wokubhala amakhodi ukuthi akhethe izisombululo ezifanele, ezibhalwe kahle kunezimbungulu kusetshenziswa iziqhathaniso zikanjiniyela
Ukushuna isitayela sokufingqa ukuze amamodeli athande izifinyezo ezimfushane, ezithembekile kunezisho noma ezihlotshaniswayo
Izingozi & Guardrails
Amaqiniso akhonjiwe angafaka ngokuthula imibiko, ukugeleza kosekelo, noma imiphumela yocwaningo.
Ukuzwela okusheshayo kungadala imiphumela engahambisani kuzo zonke izicelo ezifanayo.
Idatha yombhalo ebucayi ingase idalulwe uma izilawuli zokufinyelela zibuthakathaka.
Ukuqalisa Umhlahlandlela
Chaza ifomethi yokuphumayo, ithoni, namazinga wekhwalithi ngaphambi kokukhishwa.
Izimpendulo eziyisisekelo ngemithombo ethembekile noma nini lapho ukunemba kubalulekile.
Gcina indawo yokuhlola isibuyekezo somuntu ukuze uthole imiphumela ephezulu.
Landela amaphethini okuhluleka futhi uqeqeshe kabusha imiyalo noma ukuhamba komsebenzi njalo.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Direct Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-Odds Ratio Preference Optimization
Imibuzo evame ukubuzwa
What is Direct Preference Optimization?
I-Direct Preference Optimization (DPO) iyindlela yokuvumelanisa amamodeli olimi nokuthandwayo komuntu ngaphandle kokuqeqesha imodeli yomvuzo ehlukile noma ukuqhuba ukufunda okuqiniswayo. Ibhidliza ipayipi eliyinkimbinkimbi elinezigaba eziningi libe ukulahlekelwa ukuqeqeshwa okukodwa, okuzinzile.
Yini eqeda i-DPO uma iqhathaniswa ne-RLHF evamile?
Inzuzo eyinhloko ye-DPO ukususa imodeli yomvuzo esobala kanye ne-RL yesampula loop, ukuthuthukisa inqubomgomo ngokuqondile kumapheya athandwayo esikhundleni salokho.
Iyiphi idatha isibonelo sokuqeqeshwa se-DPO esisodwa esiqukethe?
I-DPO izitimela ngamapheya athandwayo: ukwaziswa okwengeziwe kanye neyodwa ethandwayo ('ekhethiwe') kanye nempendulo eyodwa engathandwayo ('enqatshiwe').
Iyiphi indima edlalwa imodeli yesithenjwa ku-DPO?
Ireferensi eqandisiwe (ngokuvamile imodeli ye-SFT) iqinisa ukulahlekelwa; ipharamitha ye-beta ilawula ukuthi inqubomgomo eqeqeshiwe ingasuka kude kangakanani kuyo.
Ku-DPO, umelelwa kuphi 'umvuzo'?
Ukuphuma kwe-DPO kubonisa ukuthi umvuzo usobala esilinganisweni samathuba elogi phakathi kwenqubomgomo nereferensi, ngakho-ke ayikho imodeli yomvuzo ecacile edingekayo.
Ithini ipharamitha ye-beta (izinga lokushisa) ku-DPO?
I-Beta ibusa umkhawulo ocacile we-KL: i-beta ephezulu igcina inqubomgomo iseduze nesithenjwa, i-beta ephansi ivumela ukuchezuka okwengeziwe.