Direct Preference Optimization
Direct Preference Optimization (DPO) inzira yekuenzanisa mamodheru emitauro nezvido zvevanhu pasina kudzidzisa modhi yemubairo wakasiyana kana kudzidzira kusimbisa.
Pfupiso
Iyo inodonha yakaoma yakawanda-nhanho pombi kuita imwechete, yakagadzikana kurasikirwa kwekudzidziswa.
Kudzika Kwakadzika
DPO, yakaunzwa naRafailov nevamwe vaaishanda navo kuStanford muna 2023, inofunga zvakare madzidzisiro atinoita modhi inodiwa nevanhu. Nzira yechinyakare (RLHF) inodzidzisa muenzaniso wemubairo pakuenzanisa kwevanhu, zvino inoshandisa kusimbisa kudzidza kuwedzera mubairo iwoyo. Muono wakakosha weDPO ndeyemasvomhu: iyo yakakwana mutemo pasi peiyo RLHF chinangwa ine yakavharwa-fomu hukama kune mubairo, saka iwe unokwanisa kuronga patsva maequation uye nekunatsiridza modhi yemutauro zvakananga pane zvaunofarira vaviri vaviri. Unozvipa kukurumidza, mhinduro 'yakasarudzwa' (inodiwa), uye mhinduro 'yakarambwa', uye kupatsanurwa-maitiro ekurasikirwa kunokwenya modhi kuita kuti mhinduro yakasarudzwa ive yakawanda. Hapana mubairo modhi, hapana sampling loop, hapana mubairo wekubira. Zviri nyore uye zvakanyanya kugadzikana kumhanya.
Technical Insight
DPO inoshandisa bhinari muchinjiko-entropy kurasikirwa pamusoro pezvavanoda peya. Inowedzera muyero welogi-mukana wemhinduro yakasarudzwa inoenderana neyakarambwa, imwe neimwe yakayerwa neyakaomeswa referensi modhi (kazhinji inotariswa-yakakwenenzverwa-inotangisa pekutangira). Tembiricha parameter beta inodzora kuti iyo policy ingasvike papi kubva pareferensi iyoyo, ichinyatso simbisa KL chinomanikidza icho RLHF chinoshanda zvakajeka. Mubairo wacho haumbofi wakaitwa; zviri pachena mugwaro-zvingabvira zvepolicy.
Strategic Impact
Kumhanya uye chiyero
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana.
Svika uye svika
Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana.
Sarudzo dzakajeka
Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora.
Ramangwana reKunatsiridza Zvaunofarira
DPO yave nzira yekumisikidza yekumisikidza nekuti yakachipa uye inogoneka, uye yakagadzira mhuri yezvakasiyana: IPO inogadzirisa overfitting pane-pedyo-inotemerwa zvido, KTO inodzidza kubva kune imwe chete yakanaka-kana-yakaipa mavara pachinzvimbo pevaviri, uye ORPO inopeta yekuda kudzidza mukugadzirisa zvakanaka pasina inonongedza modhi. Tarisira kuenderera mberi kwebasa rekubatanidza DPO ne-on-policy data uye kureba / kunaka debiasing, kuderedza gaka rasara neRHF yakazara yepamhepo.
Real-World Implementation
Kunyatsogadzirisa-kuvhura-huremu mamodheru ekutaura seZephyr uye akawanda Llama uye Mistral anobva kune, ayo aienderana neDPO pane zvaunofarira dataset.
Kudzikisa zvinokuvadza kana zvisingabatsiri zvinobuda uchishandisa maviri apo mhinduro yakachengeteka, inobatsira 'inosarudzwa' pane ine dambudziko.
Kudzidzisa mubatsiri wekodhi kuti asarudze mhinduro dzakaringana, dzakanyorwa zvakanaka pamusoro peiyo buggy uchishandisa dhizaini-yakatemerwa kuenzanisa.
Tuning yekupfupisa maitiro kuitira kuti mamodheru afarire pfupiso, yakatendeka pfupiso pane verbose kana idzo dzakaratidzwa.
Njodzi & Guardrails
Chokwadi chehuroyi chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana tsvakiridzo.
Kunzwa nekukasira kunogona kugadzira mhedzisiro isingaenderane pane zvikumbiro zvakafanana.
Sensitive text data inogona kuburitswa kana zvidhiraivho zvisina kusimba.
Implementation Roadmap
Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa.
Mhinduro dzepasi neakavimbika masosi pese pazvine basa.
Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda.
Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Direct Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Odds Ratio Preference Optimization
Mibvunzo inowanzo bvunzwa
What is Direct Preference Optimization?
Direct Preference Optimization (DPO) inzira yekuenzanisa mamodheru emitauro nezvido zvevanhu pasina kudzidzisa modhi yemubairo wakasiyana kana kudzidzira kusimbisa. Iyo inodonha yakaoma yakawanda-nhanho pombi kuita imwechete, yakagadzikana kurasikirwa kwekudzidziswa.
Chii chinobvisa DPO zvichienzaniswa neyechinyakare RLHF?
Mukana mukuru weDPO kubvisa iyo yakajeka yemubairo modhi uye iyo RL sampling loop, kugadzirisa iyo mutemo zvakananga pane zvaunofarira vaviri panzvimbo.
Ndeipi data iyo imwe DPO yekudzidzisa muenzaniso ine?
DPO inodzidzisa pane zvaunofarira vaviri vaviri: kukurumidza kuwedzera imwe inosarudzwa ('akasarudzwa') uye imwe isingafarirwe ('yakarambwa') mhinduro.
Ndeipi basa rinoitwa nereferenzi modhi muDPO?
Referensi yechando (kazhinji iyo SFT modhi) inosimbisa kurasikirwa; iyo beta parameter inodzora kuti mutemo wakadzidziswa unogona kubva papi kubva pairi.
MuDPO, 'mubayiro' unomiririrwa kupi?
Kutorwa kweDPO kunoratidza kuti mubairo wakajeka muyero yelogi-mukana pakati pegwaro uye referensi, saka hapana yakajeka mubairo modhi inodiwa.
Chii chinonzi beta (tembiricha) paramende muDPO kutonga?
Beta inotonga iyo KL inomanikidzirwa: yepamusoro beta inochengeta mutemo padyo nereferensi, yakaderera beta inobvumira kutsauka kwakawanda.