Basics GUIDE
Bradley-Terry Reward Modelling
Muenzaniso weBradley-Terry inzira yenhamba yezana remakore rekushandura kuenzanisa kweviri (A kurova B) kuita zvibodzwa zvenhamba.
Teerera kugwaro iriAI-yakarondedzerwa · ~2 min
Pa peji ino2 min verenga
Pfupiso
In modern AI it powers reward models that learn human preferences from 'which answer is better?' labels, the backbone of RLHF.
Kudzika Kwakadzika
Bradley-Terry, akaunzwa muna 1952, anotora chinhu chimwe nechimwe chine simba rakavanzika, uye mukana wekuti chinhu A chinorova chinhu B ibasa rekuita rekusiyana kwavo kwezvibodzwa. Mukurongeka kweAI, mepu idzi zvine hutsanana padhata rekuda: vanyoreri vevanhu vanoona mhinduro mbiri dzemuenzaniso uye vanotora iri nani, pane kupa yakaoma-ku-calibrate mhedziso zviyero. Muenzaniso wemubairo, kazhinji mutauro wemodhi ine scalar inobuda musoro, inodzidziswa kuitira kuti mhinduro inodiwa nevanhu iwane mubairo wepamusoro we scalar. Iko kurasikirwa ndiyo yakaipa log-inogona yeBradley-Terry mukana: wedzera iyo log-sigmoid ye (mubairo weakasarudzwa minus mubairo wekurambwa). Iyo inoguma yemubairo modhi yobva yawana zvibodzwa zvinobuda, ichipa chiratidzo chinosimbisa kudzidza algorithms sePPO inokwenenzvera kuita kuti mamodheru awedzere kubatsira uye anoenderana.
Kuitwa muNyika yeChokwadi
Kudzidzira iyo mubairo modhi muRLHF iyo inomira mbiri dzechatbot mhinduro uye inopa iri nani-yakaipisisa chiratidzo kuPPO-tuning yakanaka.
Direct Preference Optimization zvakanaka-tuning modhi yakananga pane yakasarudzwa-yakatarisana-yakarambwa mhinduro peya uchishandisa iyo Bradley-Terry log-sigmoid kurasikirwa.
Kuisa chess kana esports vatambi kuburikidza neElo, inova yemasvomhu hama yepedyo yeBradley-Terry modhi pane zvakabuda mumutambo.
Kuvaka chinzvimbo chekurudziro kubva ku 'vashandisi vanosarudza A pane B' tinya data pane kuyera nyeredzi.
Ramangwana reBradley-Terry Reward Modelling
Bradley-Terry anotora chinzvimbo chimwe chete chisingachinjiki uye chinoshanduka, chinoputsika kana vanhu vasingabvumirani kana kutenderera kwezvido. Tsvagiridzo iri kuenda kumhando dzinotora kugovaniswa kwekuda, mibairo yemhando dzakasiyana-siyana (rubatsiro, chengetedzo, kutendeseka kunopihwa zvakasiyana), uye nzira dzakaita seNash kudzidza kubva kumhinduro dzevanhu dzinodonhedza fungidziro yechikamu chimwe chete. DPO uye akasiyana ayo anowedzera kupeta chinangwa cheBradley-Terry zvakananga mukudzidziswa kwepolicy. Tarisira zvirongwa zvekuenzanisa zvakapfuma, zvinosanganisira marongero ezvinopfuura zvinhu zviviri uye zvinoyemurika-zviyero zvaunofarira, kuderedza mibairo yekubira.
Kuzvishandisa
Nei zvichikosha
- Sarudzo dzakajeka. Inokubatsira kuparadzanisa zvakajeka zvichemo zvehunyanzvi kubva mumutauro wekushambadzira.
- Mutengo uye bhajeti. Iwe unogona kubvunza zvirinani kuita mibvunzo usati washandisa mari kana nguva.
- Team uye workflow. Zvikwata zvine nzwisiso yakagovaniswa inoita zvirinani chigadzirwa, mutemo, uye sarudzo dzekudzidza.
Chenjera
- Zvikwata zvakasiyana zvinogona kushandisa izwi rimwechete zvakasiyana, saka tsanangura nzvimbo nekukurumidza.
- Benchmarks inogona kutaridzika yakasimba nepo chaiyo-yenyika kuita isina kuenzana.
- Kuregeredza mhando yedata uye zvirongwa zvekuongorora zvinowanzogadzira mhedzisiro isina kusimba.
Matanho anotevera
- Tanga netsanangudzo yemutauro wakajeka yemhedzisiro yaunoda.
- Sarudza metric imwe yekubudirira uye imwe yekutadza mamiriro usati waedzwa.
- Mhanya mutyairi mudiki ane data remumiriri, kwete demo rakakwenenzverwa.
- Gwaro uko Bradley-Terry Reward Modelling inobatsira uye uko nzira dzakareruka dziri nani.
Ramba Uchiongorora
Tsamba yemahara
Tora iyo yemazuva ese AI pfupiso
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Take the Bradley-Terry Reward Modeling quiz
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Mibvunzo inowanzo bvunzwa
What is Bradley-Terry Reward Modeling?
Muenzaniso weBradley-Terry inzira yenhamba yezana remakore rekushandura kuenzanisa kweviri (A kurova B) kuita zvibodzwa zvenhamba. MuAI yemazuva ano inopa mibairo mienzaniso inodzidza zvido zvevanhu kubva 'ndeipi mhinduro iri nani?' mavara, musana weRLHF.
Chii chinoshandura modhi yeBradley-Terry kuita zvibodzwa zvenhamba?
Bradley-Terry anotora paviri 'A beats B' kuenzanisa uye anoisa yakavanzwa simba mamaki echinhu chimwe nechimwe, ndosaka ichienderana nezvinodiwa nevanhu kunyora zvakanyanya.
MuBradley-Terry, mukana wekuti A anorova B ibasa rei?
Iyo modhi inoseta P(A inorova B) yakaenzana neiyo logistic (sigmoid) yemusiyano pakati peA's uye B yakavanzwa simba zvibodzwa.
Sei mibairo yeBradley-Terry ichizivikanwa chete kusvika kune yekuwedzera nguva dzose?
Kurasikirwa kwekudzidzira kunoshandisa chete mutsauko pakati pemibairo yakasarudzwa uye yakarambwa, saka kushandura zvibodzwa zvese neyakafanana nguva dzose kunosiya kurasikirwa kusingachinji; chikero chakakwana hachidi.
Ndeipi kurasikirwa kwakajairwa kwekuenzanisa kumwe chete kwekuda mumuenzaniso wemubairo?
Iyo Bradley-Terry isina kunaka log-inogona inoderedza kusvika pakubvisa log-sigmoid yemusiyano wemubairo wakasarudzwa-minus-wakaramba, kusundira mubairo wemhinduro yakasarudzwa kumusoro.
Ndeipi nzira inosvetuka yakaparadzana mubairo modhi nekugadzirisa iyo Bradley-Terry chinangwa chakanangana nepolicy?
DPO inogadziridza chinangwa cheBradley-Terry chekuda kuitira kuti chishandiswe zvakananga kumuenzaniso wemitemo, kubvisa kudiwa kwekudzidzisa uye kubvunza muenzaniso wemubairo wakamira.
Ramba uchidzidza
Mazano anoenderana
Mamwe madhairekitori akasarudzirwa nyaya iyi