Optimisation taamu direct
Direct Preference Optimization (DPO) anam la wu ñuy méngale xeeti làkk yi ak tànneefi nit ñi te duñu tàggat benn xeetu neexal bu wuute wala jàng buy dooleel.
Résumé
Dafay dindi gasoduc bu bari etap ci benn ñàkka tàggat bu dëgër.
Plongeur bu xóot
DPO, Rafailov ak ay naataangoom ñoo ko dugal ci Stanford ci 2023, mu xalaat ci ni ñuy jàngalee xeetu li nit ñi taamu. Xeetu cosaan (RLHF) dafay tàggat xeetu neexal ci méngale nit, ba noppi jëfandikoo jàng buy dooleel ngir yokk neexal googu. DPO gis-gis bu am solo bi mooy math: politik bi gëna baax ci suufu mébetu RLHF amna diggante bu tëju ak neexal bi, suko defee nga mëna defaraat equation yi te gëna mëna jëfandikoo modelu làkk wi ci ñaari tànneef. Danga koy jox ab laaj, tontu bu 'tann' (li taamu), ak tontu bu 'bañ', ak benn xeetu xaaj bu yomb buy nudge model bi ngir def tontu bu tann bi gëna am wërsëg. Amul xeetu neexal, amul bouclage échantillonnage, amul pirate neexal. Dafa gëna yomba daw te gëna dëgër.
Gis-gis xarala
DPO dafay jëfandikoo entropi croix binar buy ñàkk ci kaw ñaari tànneef. Dafay yokk ratio bi log-probabilite ci tontu biñ tànn méngoo ak biñ bañ, bu nekk ci ñoom ñu natt ko ci xeetu royuwaay buñ gelé (dafay faral di nekk point de départ buñ xool bu baax). Benn parametru tàngoor beta mooy saytu ba fu politik bi mëna dem fu sori ci royuwaay boobu, di jaay doole KL bi RLHF di jëfandikoo ci anam wu leer. Neexal bi du musa am; Dafa nekk ci log-probabilite yi ci politik bi.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëg gëna xéewale taamu direct
DPO nekkaatna anam wuñ jagleel ndax yomb na te mën nañu ko defaraat, ba noppi jural na famiy yu bari: IPO dafay defar overfitting ci tànneef yu jege-deterministic, KTO dafay jàng ci benn etiket bu baax wala bu baaxul ci barabu ñaar, ORPO dafay jàng tànneef ci xeetu fine-tuning bu amul royuwaay. Xaarandi liggéey bu wéy ci boole DPO ak done ci politik ak guddaay / kalite debiasing, wàññi bërëb bi des ak RLHF bu mat sëkk ci net bi.
Doxal ci àdduna dëgg
Xeetu waxtaan yu ubbeeku yu melni Zephyr ak yeneen Llama ak Mistral, yuñ boole ak DPO ci setu done yiñ taamu
Wàññi li génne luy lore wala lu amul njariñ ci jëfandikoo ñaar fu tontu bu wóor, am njariñ lañu 'tànn' ci kaw tontu bu am jafe-jafe
Jàngale assistant codage mu taamu pexe yu jaar yoon, yu am këyit yu baax, moo gën yu am ay njuumte, jëfandikoo ay méngale yu developpeur yi joxe
Tuning stil resumé suko defee model yi gëna bëgg resumé yu gàtt te wóor, moo gën yu bari wax wala yu xawa réer
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Direct Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Optimisation Ratio Odds
Laaj yi ñuy faral di laaj
Luy gëna xéewale taamu direct?
Direct Preference Optimization (DPO) anam la wu ñuy méngale xeeti làkk yi ak tànneefi nit ñi te duñu tàggat benn xeetu neexal bu wuute wala jàng buy dooleel. Dafay dindi gasoduc bu bari etap ci benn ñàkka tàggat bu dëgër.
Lan la DPO dindi buñu ko méngale ak RLHF cosaan?
Njariñu DPO bi gëna mag mooy dindi xeetu neexal bu leer bi ak bouclage sampling RL, gëna xéewale politik bi ci ñaari tànneef yi.
Ban done la benn misaalu tàggat DPO am?
DPO dafay tàggat ci ñaari tànneef: ab laaj boole ci benn tontu buñu taamu ('tann') ak benn tontu buñu taamuwul ('bañ').
Ban cër la xeetu royuwaay bi di def ci DPO?
Royuwaay bu liw (li gëna bari ci xeetu SFT) mooy ancre perte bi; parametru beta bi mooy saytu ba fu politik biñ tàggat mëna toxu ci moom.
Ci DPO, fan lañuy wane 'neexal' bi?
Derivation DPO dafay wane ni neexal bi dafa nekk ci ratio bi log-probability diggante politik ak royuwaay, kon soxlawul xeetu neexal bu leer.
Lan mooy parametru beta (tamperatiir) bi ci DPO bi?
Beta mooy doxal tënk bu nëbbu bi ci KL: beta bu gëna kawe dafay tëye politik bi gëna jege royuwaay bi, beta bu gëna suufe dafay may deviation bu gëna bari.