Bradley-Terry neexal
Modèle Bradley-Terry ab pexem lim la bu yàgg lool ngir soppi méngale ñaar (A dafa raw B) ci poñ yu am lim.
Résumé
In modern AI it powers reward models that learn human preferences from 'which answer is better?' labels, the backbone of RLHF.
Plongeur bu xóot
Bradley-Terry, miñu dugal ci atum 1952, dafa jàpp ni mbir mu nekk amna poñ bumu nëbb ci dooleem, te ni mbir A di raw mbir B mooy fonction logistic bu seen wuute ci poñ yi. Ci IA alignment, lii dafay méngoo bu baax ak done yiñ taamu: etiketu nit ñi dañuy gis ñaari tontu model ba noppi tànn ki gëna baax, ludul joxe note absolu yu jafe kalibre. Benn xeetu neexal, lu ci gëna bari mooy xeetu làkk wi am boppu génne scalar, dañu koy tàggat ngir tontu bi nit ñi taamu am neexal scalar bu gëna mag. Perte bi mooy log-likelihood bu baaxul ci probabilite Bradley-Terry: yokk log-sigmoid bu (neexal biñ tànn dindi neexal biñ bañ). Modèlu neexal biy génne dafay am poñ ci génnug arbitraire, di joxe siñaal bi algorithm jàng buy dooleel lu melni PPO optimize ngir defar model yu gëna am njariñ ak gëna dëppoo.
Gis-gis xarala
Perte tàggat yaram ngir méngale lu yomb la ci dindi log-sigmoid bu (r_chosen - r_rejected), kon model bi du jàng ludul wuute relatif. Loolu dafay tekki ni neexal yi mën nañu leen xàmmee fileek amul benn constant buy yokk; balans absolu bi amul benn njariñ. Ndax méngale yi ñoo gëna yomba te gëna dëppoo ci nit ñi moo gën 1 ba 10 poñ, done yu Bradley-Terry ñoo gëna néew bruit. Taamu direct bi gëna xéewale ginaaw ga dafa wane ni mën nga teggi xeetu neexal bu wuute bi te gëna xéewale mébetu Bradley-Terry ci saasi ci politik bi.
njeextalu pexe
dogal yu gëna leer
Daf lay jàppale nga tàqale kàddu yu leer ci wàllu xarala ak làkku fësal njaay.
Njëgg ak budget
Mën nga laaj laaj yu gëna baax ci samp gi balaa ngay dugal xaalis wala sa jotu liggéey.
Ekip ak def liggéey
Ekip yi bokk xam-xam ñoo gëna mëna jël yenn dogal ci wàllu produit, politik ak jàng.
Ëlëgu xeetu neexal bu Bradley-Terry
Bradley-Terry dafay xalaat benn rang bu méngoo ak tànneef yuy jàll, yuy yàqu su nit ñi déggoowul wala tànneef yi di soppiku. Gëstu baa ngi jubal ci xeetu séddale yiñ taamu, neexal yu bari (ndimbal, kaaraange, njub ñu tànn leen), ak pexe yu melni Nash jàng ci feedback nit ñi di daaneel xalaatu benn poñ. DPO ak ay xeetam dañuy gëna boole mébetu Bradley-Terry ci tàggat ci wàllu politik. Xaarandil ay pexe méngale yu gëna riis, yu ci melni rang yu ëpp ñaari mbir ak tànneef yu lalu ci wóolu sa bopp, ngir wàññi pirateri neexal yi.
Doxal ci àdduna dëgg
Taggat xeetu neexal ci RLHF bi rang ñaari tontu chatbot ak joxe siñaal bi gëna baax-baaxul ci PPO fine-tuning.
Taamu bu jub bi gëna xéewale defar ab xeetu tontu bu jub ci tanneef-ak-bañ ñaari tontu yu jëfandikoo Bradley-Terry log-sigmoid perte.
Ranking joueur echec wala esports jaaraleko ci Elo, muy benn cousin bu jege ci modelu Bradley-Terry ci njariñu jeu bi.
Tabax ab rang buy xelal ëmbiit li juge ci 'jëfandikukat yi taamu A moo gëna B' klike ci done yi moo gën notu biddiiw yu mat sëkk.
Risk yi ak balustrade yi
Ekip yu bari mën nañu jëfandikoo benn baat ci anam wu wuute, kon teela leeral yaatuwaayam.
Benchmark yi mën nañu nuru lu am doole waaye performance yi ci àdduna bi duñu tolloo.
Bëgg kalite done ak palaŋu jàngat dafay faral di jur njariñ yu yomba dagg.
Roadmap ngir samp gi
Tàmbaleel ci joxe leeral ci làkk wu leer ci njariñ li nga soxla.
Tannal benn metric bu baax ak benn anam bu baaxul balaa ngay saytu.
Doxal ab pilote bu ndaw ak ay done yu representatif, du ab demo bu leer.
Bindal fi Bradley-Terry Neexal Modeling di jàppale ak fi pexe yu gëna yomba gëna baax.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Bradley-Terry Reward Modeling quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Modeling neexal
Laaj yi ñuy faral di laaj
What is Bradley-Terry Reward Modeling?
Modèle Bradley-Terry ab pexem lim la bu yàgg lool ngir soppi méngale ñaar (A dafa raw B) ci poñ yu am lim. Ci juntuwaay yu bees yi defay dooleel ay xeetu neexal yuy jàng tànneefi nit ci 'ban tontu moo gën?' etiket, yaxu ndigg RLHF.
Lan la xeetu Bradley-Terry di soppi ci poñ yi?
Bradley-Terry dafay jël ñaar-ñaar 'A daan B' méngale ba noppi génne poñ doole bu nëbbu ci mbir mu nekk, moo tax mu méngoo bu baax ak tànneefi nit ñi.
Ci Bradley-Terry, lu A mëna raw B mingi aju ci lan?
Modèle bi dafay tëral P(A dafay dóor B) mu tollu ci logistic (sigmoid) bu wuute gi am ci digganté poñ yi A ak B am ci doole yu nëbbu yi.
Lan moo waral neexal yu Bradley-Terry mën nañu leen xàmmee fileek amul benn coppite buy yokk?
Perte ci tàggat yaram dafay jëfandikoo ludul wuute gi nekk ci diggante neexal yiñ tànn ak yiñ bañ, kon sooy soppi poñ yépp ci benn constant, perte bi du soppiku; balans absolu bi amul benn njariñ.
Lan mooy perte standard ngir benn tànneef ci modelu neexal?
Bradley-Terry log-likelihood bu baaxul bi dafay wàññi ba minus log-sigmoid ci tànneef-minus-bañ wuute neexal, push neexal tontu biñ tànn mu gëna kawe.
Ban anam mooy jël xeetu neexal bu wuute ci gëna xéewale mébetu Bradley-Terry ci kaw politik bi?
DPO dafay soppi mébetu tànneefu Bradley-Terry suko defee ñu mëna ko jëfandikoo ci xeetu politik bi, dindi soxla tàggat ak laaj xeetu neexal bu nekk ci boppam.