Nwelite Mmasị Ratio ọdịghị mma
Odds Ratio Preference Optimization (ORPO) bụ usoro nlegharị anya nke ọma na-akuzi ụdị omume ọma na mmasị mmadụ n'otu ngafe ọzụzụ.
Nchịkọta
It matters because it skips the usual separate reward model and reference model, making alignment cheaper and simpler.
Ime miri emi
ORPO, nke Hong, Lee na Thorne webatara n'afọ 2024, na-ejikọta nhazigharị ọma na ntinye mmasị n'ime otu nzọụkwụ. Ọtụtụ pipeline alignment na-ebu ụzọ mee SFT na ezigbo ọmụmaatụ, wee mee usoro nke abụọ dị ka RLHF ma ọ bụ DPO nke chọrọ nnomi oyi oyi nke ụdị (ntụaka) gbakwunyere ụzọ abụọ mmasị echekwara. ORPO na-ewepụ ụdị ntụaka kpamkpam. Ọnwụ ya na-agbakwụnye okwu ntaramahụhụ n'ụkpụrụ na-esote akara ebumnobi: ọ na-ebuli ohere nke ihe nlereanya na-enye nzaghachi nke ahọpụtara (ọ masịrị) ka ọ na-eweda ohere nke onye jụrụ ajụ. N'ihi na ọ na-eji oke erughị eru kama inwe ọdịiche log-probability siri ike, ntaramahụhụ ahụ dị nro, yabụ ihe nlereanya ahụ na-amụta ịkwado azịza dị mma na-echefughị ọgbọ nwere ọfụma.
Nghọta nka nka
Ọnwụ ORPO bụ SFT cross-entropy mfu gbakwunyere log-sigmoid dị arọ nke oke log odds n'etiti nzaghachi ahọpụtara na jụrụ. Odds nhata p / (1-p), yabụ nha nha na-atụnyere etu ihe nlereanya ahụ ga-esi nweta azịza dị mma yana nke ọjọọ. Iji ọgbaghara kama ịbụ ihe puru omume na-eme ka ọdịiche ahụ dị nro, nke na-egbochi oke mkpochapụ nke akara ngosi ndị a jụrụ ajụ nke nwere ike iweda ụdị enweghị atụ.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke Mmasị Mmasị Ọdịmma
ORPO na-enweta traction n'ihi na ọ na-ebelata ebe nchekwa ma gbakọọ site na idobe usoro ntụnye aka, nke mara mma maka otu egwuregwu na-emezigharị na ngwaike nwere oke. Na-atụ anya na ọ ga-apụta ugboro ugboro na ntụziaka ndị mepere emepe yana dị ka nhọrọ ndabara na ụlọ akwụkwọ dị ka Hugging Face TRL. Ọ ga-abụ na ọrụ ga-eme n'ọdịnihu na-emegharị ịdị arọ nke lambda na-akpaghị aka, jikọta ORPO na ebumnuche ndị ọzọ na-enweghị ntụaka, wee gbasaa ya na ọtụtụ ụdị na nnukwu ụdị ebe ijide akwụkwọ abụọ na ebe nchekwa dị oke ọnụ.
Mmejuputa n'ezie n'ụwa
Idozi ụdị nkata 7B mepere emepe na ụzọ abụọ mmasị na-ebugo akwụkwọ ntụaka nke abụọ, na-ebelata ebe nchekwa GPU.
Mmalite na-akwado onye na-enyere ndị ahịa aka ịhọrọ nkwanye ùgwù na azịza amụma n'otu ọsọ ọzụzụ kama SFT-mgbe ahụ-DPO
Ndị nyocha na-atụnyere ORPO megide DPO n'otu dataset iji gosi ndakọrịta yiri ya na kọmpụta dị ala.
Ịmegharị ụkpụrụ ntọala na ngalaba pụrụ iche (dịka ọmụmaatụ, idepụta iwu) ebe ụzọ abụọ dị mma na nke ọjọọ dị mana mmefu ego-ụgwọ ọrụ abụghị
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Odds Ratio Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nkwalite mmasị ozugbo
Ajụjụ a na-ajụkarị
What is Odds Ratio Preference Optimization?
Odds Ratio Preference Optimization (ORPO) bụ usoro nlegharị anya nke ọma na-akuzi ụdị omume ọma na mmasị mmadụ n'otu ngafe ọzụzụ. Ọ dị mkpa n'ihi na ọ na-amapụ ụdị ụgwọ ọrụ dị iche iche yana ụdị ntụaka, na-eme ka nhazi ya dị ọnụ ala ma dị mfe.
Kedu uru bụ isi bara uru nke ORPO karịa usoro dịka DPO?
ORPO na-atụgharị mmasị mmụta n'ime mfu SFT na ọ dịghị mkpa nnomi nrụtụ aka oyi kpọnwụrụ nke ihe nlereanya, na-echekwa ebe nchekwa na gbakọọ.
Kedu ihe 'oke nha' dị na ORPO tụnyere?
ORPO na-eji oke nke erughị eru (p/(1-p)) ihe nlereanya na-ekenye azịza ahọrọ na nke achọghị.
ORPO na-arụ ọrụ abụọ n'ime otu nkuzi gafere?
ORPO na-ejikọta ọkọlọtọ SFT na-esote-token ebumnobi na ntaramahụhụ mmasị na otu mfu jikọtara ọnụ.
Kedu ihe kpatara ORPO ji eji ọgbaghara karịa ka ọ dị iche iche nwere ike itinye akwụkwọ maka ntaramahụhụ ahụ?
Ntaramahụhụ dabere n'ọgbaghara dị ntakịrị, na-enyere ihe nlereanya na-enweghị atụ aka ịnọgide na-enwe ọgbọ mara mma ma ọ ka na-ahọrọ azịza dị mma.
Akọwara mfu ORPO kacha mma dị ka nke ngwakọta?
ORPO na-agbakwụnye okwu oke log-sigmoid odds-ratio n'elu mfu-entropy a na-elekọta.