Ukwenqaba Isampula Ukushuna Kahle
I-Rejection Sampling Fine-Tuning (RFT) ikhiqiza izimpendulo eziningi zekhandidethi, igcina kuphela ezinamagoli amahle kakhulu, futhi iphinda iqeqeshe imodeli kulabo abawinile.
Uhlolojikelele
It matters because it offers much of RLHF's benefit using straightforward supervised learning instead of complex reinforcement learning.
I-Deep Dive
Ukwenqaba Isampula Ukushuna Kahle, kwesinye isikhathi okubizwa ngokuthi ukuhlelwa kahle kwe-best-of-N, kuyisithako esiyinhloko endleleni amamodeli afana ne-Meta's Llama 2 kanye ne-Llama 3 aqondaniswe ngayo. Iresiphi ilula: ngokwaziswa ngakunye, sampula izimpendulo ezimbalwa (ithi 4 kuye ku-64) kumodeli yamanje, phawula ngayinye ngemodeli yomklomelo noma ishekha esizenzakalelayo, bese ulahla ('yenqaba') konke ngaphandle kwemiphumela eseqophelweni eliphezulu. Amasampuli asele ekhwalithi ephezulu aba isethi yedatha yokushuna kahle entsha egadiwe, futhi imodeli iqeqeshwa kuwo ngokulahleka kwethokheni elandelayo okuvamile. Ukuphinda le loop kugudluza imodeli ekukhiqizeni izimpendulo ezingcono iyodwa. Ngenxa yokuthi imodeli ifunda kokuphumayo okuhlungiwe, i-RFT igwema ukungazinzi nokushuna ikhanda le-RL-gradient RL kuyilapho isasebenzisa isignali yomvuzo.
I-Technical Insight
I-RFT isebenzisa iqiniso lokuthi ukwenza amasampula izikhathi eziningi nokugcina impendulo yomvuzo omkhulu kulinganiselwa ukukhetha ekusabalaliseni okucijile, kwekhwalithi ephezulu. Ukuqeqeshwa kulabo abawinile kusetshenziswa i-cross-entropy ejwayelekile kukhipha kahle ukuziphatha okuhle kwe-N emuva kokuphumayo kwesampula eyodwa yemodeli. Ezizindeni eziqinisekisiwe njengezibalo noma ikhodi, 'umvuzo' ungase ube ukuthi impendulo yokugcina noma ukuhlolwa kweyunithi kuyaphumelela, kususe isidingo semodeli yomklomelo ofundiwe ngokuphelele.
I-Strategic Impact
Isivinini nesikali
Ukugeleza komsebenzi wolimi kungahamba ngokushesha ngaphandle kokudela ukuvumelana.
Finyelela futhi ufinyelele
Yandisa ukufinyelela kuzo zonke izilimi nezitayela zokuxhumana.
Izinqumo ezicacile
Amaqembu angachitha isikhathi esiningi ekwahluleleni kuyilapho i-automation isingatha impinda.
Ikusasa Lokwenqatshwa Kwesampula Ukushuna Kahle
I-RFT iwumgogodla wokuqeqeshwa kwangemuva kwesimanje, evame ukusetshenziswa ngaphambi noma eduze nezindlela ze-RL njenge-PPO ne-DPO. Ukukhanga kwayo kukhula ngemibono eshibhile kanye neziqinisekiso ezizenzakalelayo eziqinile: njengoba amamodeli eba ngcono ekuzikhiqizeni nasekuzihloleni, amasampula okunqatshelwa okuphindaphindiwe asekela idatha yokwenziwa kanye nezihibe zokuzithuthukisa. Lindela ukuhlanganiswa okuqinile namamodeli okucabanga akhiqiza amaketango emicabango angaqinisekiswa, kanye nocwaningo oluqhubekayo lokuthi ungagwema kanjani ukugetshengwa kwemivuzo kanye nokuwa kwezinhlobonhlobo lapho uziqeqesha ngokuphindaphindiwe emiphumeleni yemodeli ngokwayo.
Ukuqaliswa Komhlaba Wangempela
Ukuqondanisa amamodeli esitayela se-Llama ngokuthatha isampula izimpendulo eziningi ngesikhathi ngasinye, ngokugcina amaphuzu aphezulu emodeli yomvuzo, bese kuba yi-SFT kulawo
Ukuthuthukisa isixazululi sezibalo ngokuveza izixazululo eziningi nokugcina kuphela lezo ezifinyelela impendulo efanele, ehlolekayo.
Ukukhiqizwa kwekhodi lapho amakhandidethi agcinwa kuphela uma ephumelele ukuhlolwa kweyunithi, bese esetshenziswa njengedatha yokuqeqeshwa
Ukwakha amasethi edatha emiyalo yokwenziwa ngokuhlunga izimpendulo ezizikhiqizela yona ezihamba phambili zemodeli emzuliswaneni olandelayo wokuqeqeshwa.
Izingozi & Guardrails
Amaqiniso akhonjiwe angafaka ngokuthula imibiko, ukugeleza kosekelo, noma imiphumela yocwaningo.
Ukuzwela okusheshayo kungadala imiphumela engahambisani kuzo zonke izicelo ezifanayo.
Idatha yombhalo ebucayi ingase idalulwe uma izilawuli zokufinyelela zibuthakathaka.
Ukuqalisa Umhlahlandlela
Chaza ifomethi yokuphumayo, ithoni, namazinga wekhwalithi ngaphambi kokukhishwa.
Izimpendulo eziyisisekelo ngemithombo ethembekile noma nini lapho ukunemba kubalulekile.
Gcina indawo yokuhlola isibuyekezo somuntu ukuze uthole imiphumela ephezulu.
Landela amaphethini okuhluleka futhi uqeqeshe kabusha imiyalo noma ukuhamba komsebenzi njalo.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Rejection Sampling Fine-Tuning quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-QLoRA kanye ne-4-Bit Fine-Tuning
Imibuzo evame ukubuzwa
What is Rejection Sampling Fine-Tuning?
I-Rejection Sampling Fine-Tuning (RFT) ikhiqiza izimpendulo eziningi zekhandidethi, igcina kuphela ezinamagoli amahle kakhulu, futhi iphinda iqeqeshe imodeli kulabo abawinile. Kubalulekile ngoba inikeza okuningi kwezinzuzo ze-RLHF ngokusebenzisa ukufunda okugadiwe okuqondile esikhundleni sokufunda okuyinkimbinkimbi.
Uyini umbono oyinhloko weRejection Sampling Fine-Tuning?
I-RFT isampula izimpendulo eziningi, izihlungi kwabanamaphuzu aphezulu, futhi ishuna kahle imodeli kulabo abawinile.
Ezizindeni ezingaqinisekiswa njengezibalo noma ikhodi, yini engasebenza njengomvuzo?
Ngemisebenzi ehlolekayo, i-RFT ingasebenzisa ukufaneleka kwangempela noma ukuhlolwa kweyunithi, igweme imodeli yomvuzo efundiwe.
Imuphi umndeni oyimodeli odumile owasebenzisa isampula yokulahla ngesikhathi sokuqondanisa?
Meta kuchazwe kusetshenziswa amasampula okwenqaba njengengxenye yeresiphi yangemuva kokuqeqeshwa yamamodeli e-Llama 2 kanye ne-Llama 3.
Kungani amaqembu engase akhethe i-RFT kune-RL-gradient RL njenge-PPO?
I-RFT iphinda iziqeqeshe ngokulahlekelwa kwethokheni elandelayo kumasampuli ahlungiwe, ishiya ubunzima bokushuna kanye nokungazinzi kwezindlela zenqubomgomo-gradient.
Ukuqeqeshwa ngamasampula omvuzo omkhulu kwenzani ngempumelelo?
Ngokufunda ezimpendulweni ezihlungiwe eziphezulu, imodeli ifaka ngaphakathi ukuziphatha kwekhwalithi ephezulu esizukulwaneni esisodwa.