Guhitamo neza
Ibyifuzo bya Optimisation itaziguye (DPO) nuburyo bwo guhuza imiterere yururimi nibyifuzo byabantu utabanje guhugura ibihembo bitandukanye cyangwa gukora imyitozo yo gushimangira.
Incamake
It collapses a complex multi-stage pipeline into a single, stable training loss.
Kwibira cyane
DPO, yatangijwe na Rafailov na bagenzi be i Stanford mu 2023, irongera itekereza uburyo twigisha icyitegererezo ibyo abantu bakunda. Uburyo gakondo (RLHF) butoza icyitegererezo cyigihembo kubigereranya ryabantu, hanyuma bigakoresha kwiga gushimangira kugirango ibihembo byinshi. Ubushishozi bwibanze bwa DPO ni imibare: politiki nziza iyobowe nintego ya RLHF ifitanye isano ifunze nigihembo, urashobora rero gutondekanya ibigereranyo hanyuma ugahindura imiterere yururimi muburyo bubiri. Uratanga igisubizo, igisubizo 'cyatoranijwe' (gikunzwe), hamwe nigisubizo 'cyanze', hamwe nuburyo bworoshye bwo gutondekanya-uburyo bwo gutakaza icyerekezo cyerekana igisubizo cyatoranijwe bishoboka. Nta bihembo byigihembo, nta sample loop, nta bihembo bya hacking. Nibyoroshye cyane kandi bihamye gukora.
Ubushishozi
DPO ikoresha binary cross-entropy igihombo kuruta guhitamo byombi. Yongera igipimo-gishoboka cyo kugereranya igisubizo cyatoranijwe ugereranije nicyo cyanze, buriwese yapimwe nicyitegererezo cyakonjeshejwe (mubisanzwe bigenzurwa-neza-neza neza aho byatangiriye). Ubushyuhe bwa beta bugenzura intera politiki ishobora kuva kuri iyo nyandiko, igashyira mu bikorwa mu buryo butaziguye imbogamizi ya KL RLHF ikoresha mu buryo bweruye. Igihembo nticyigera kibaho; biragaragara muri politiki yonyine log-probabilite.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Igihe kizaza cyo guhitamo neza
DPO yahindutse uburyo budasanzwe bwo guhuza kuko bihendutse kandi byororoka, kandi byabyaye umuryango wibintu bitandukanye: IPO ikosora birenze urugero kubyo ukunda-kugena ibintu, KTO yigira kumurongo umwe mwiza-cyangwa-mubi aho kuba babiri, kandi ORPO ikunda kwiga mukuringaniza neza nta moderi yerekana. Tegereza imirimo ikomeza yo guhuza DPO namakuru ya politiki hamwe nuburebure / ubuziranenge bwa debiasing, kugabanya icyuho gisigaye hamwe na RLHF yuzuye kumurongo.
Gushyira mu bikorwa Isi
Kuringaniza neza uburyo bwo kuganira buremereye nka Zephyr hamwe nibikomoka kuri Llama na Mistral byinshi, byahujwe na DPO kumibare yamakuru.
Kugabanya ibisubizo byangiza cyangwa bidafasha ukoresheje bibiri aho igisubizo cyizewe, gifasha 'cyatoranijwe' hejuru yikibazo
Kwigisha umufasha wa code guhitamo guhitamo neza, byanditse neza ibisubizo kurenza amakosa ukoresheje igereranya ryakozwe nabaterankunga
Guhuza incamake yuburyo rero icyitegererezo gikunda incamake, kwizerwa kurenza verose cyangwa salusiyo
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Direct Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Ikigereranyo cya Odds Ibyifuzo byo guhitamo
Ibibazo bikunze kubazwa
What is Direct Preference Optimization?
Ibyifuzo bya Optimisation itaziguye (DPO) nuburyo bwo guhuza imiterere yururimi nibyifuzo byabantu utabanje guhugura ibihembo bitandukanye cyangwa gukora imyitozo yo gushimangira. Isenyuka umuyoboro utoroshye wibyiciro byinshi murwego rwo gutakaza imyitozo ihamye.
Niki DPO ikuraho ugereranije na RLHF gakondo?
Inyungu nyamukuru ya DPO ni ugukuraho ibihembo byerekana ibihembo hamwe na RL y'icyitegererezo, guhitamo politiki muburyo bubiri aho guhitamo.
Ni ayahe makuru urugero rumwe rwa DPO rugizwe?
DPO ihugura kubintu byombi: ikibazo wongeyeho kimwe cyatoranijwe ('cyatoranijwe') hamwe nigisubizo kitemewe ('cyanze').
Ni uruhe ruhare icyitegererezo cyerekana muri DPO?
Icyerekezo cyakonjeshejwe (mubisanzwe icyitegererezo cya SFT) cyerekana igihombo; ibipimo bya beta bigenzura intera politiki yatojwe ishobora kuva muri yo.
Muri DPO, 'ibihembo' bigaragarira he?
Inkomoko ya DPO yerekana ibihembo bitagaragara mu kigereranyo cya log-probabilite hagati ya politiki n’ibisobanuro, bityo rero nta buryo bwo gutanga ibihembo bukenewe.
Niki ibipimo bya beta (ubushyuhe) muri DPO igenzura?
Beta igenga imbogamizi ya KL itagaragara: beta yo hejuru ituma politiki yegera aho yerekeza, beta yo hasi yemerera gutandukana.