Haɓaka fifiko kai tsaye
Haɓaka fifiko kai tsaye (DPO) hanya ce ta daidaita ƙirar harshe tare da abubuwan son ɗan adam ba tare da horar da keɓantaccen samfurin lada ko gudanar da koyo na ƙarfafawa ba.
Dubawa
It collapses a complex multi-stage pipeline into a single, stable training loss.
Zurfafa nutsewa
DPO, wanda Rafailov da abokan aiki suka gabatar a Stanford a cikin 2023, ya sake tunanin yadda muke koyar da samfurin abin da mutane suka fi so. Hanyar gargajiya (RLHF) tana horar da samfurin lada akan kwatancen ɗan adam, sannan yayi amfani da koyon ƙarfafawa don haɓaka wannan lada. Mabuɗin fahimtar DPO na lissafi ne: ingantacciyar manufa a ƙarƙashin wannan manufar RLHF tana da alaƙa mai rufaffiyar tsari ga lada, saboda haka zaku iya sake tsara ma'auni da haɓaka ƙirar harshe kai tsaye akan nau'ikan zaɓin zaɓi. Kuna ba shi gaggauwa, amsa 'zaɓaɓɓe' (wanda aka fi so), da martanin 'wanda aka ƙi', da sauƙaƙan nau'in rarrabuwa yana nudge samfurin don sanya amsar da aka zaɓa ta fi dacewa. Babu samfurin lada, babu madauki samfurin, babu satar lada. Ya fi sauƙi kuma mafi kwanciyar hankali don gudu.
Fahimtar Fasaha
DPO yana amfani da asarar giciye-entropy na binary akan nau'ikan zaɓin zaɓi. Yana haɓaka rabon yuwuwar log na zaɓin amsa dangane da wanda aka ƙi, kowanne an auna shi da daskararrun ƙirar ƙira (yawanci wurin farawa mai kyau-mai kyau-saurare). Ma'aunin zafin jiki na beta yana sarrafa yadda manufar zata iya yin nisa daga waccan ma'anar, a fakaice tana aiwatar da ƙaƙƙarfan KL wanda RLHF ke aiki a sarari. Ba a taɓa samun lada; yana cikin fayyace a cikin yuwuwar rajistar manufofin.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar Inganta fifikon Kai tsaye
DPO ya zama hanyar daidaitawa ta asali saboda yana da arha kuma ana iya sakewa, kuma ya haifar da dangi na bambance-bambancen: IPO yana gyara overfitting akan abubuwan da ake so na kusa, KTO yana koyo daga lakabi mai kyau-ko-mara kyau maimakon nau'i-nau'i, kuma ORPO yana ninka fifikon koyo cikin daidaitawa mai kyau ba tare da wani samfurin tunani ba. Yi tsammanin ci gaba da aiki akan haɗa DPO tare da bayanan kan-manufa da tsawaitawa / inganci, rage ragowar rata tare da cikakken RLHF akan layi.
Aiwatar da Gaskiyar Duniya
Kyakkyawar ƙirar taɗi mai buɗewa mai nauyi kamar Zephyr da yawancin abubuwan Llama da Mstral, waɗanda suka daidaita tare da DPO akan abubuwan da aka zaɓa.
Rage abubuwan da ke cutarwa ko mara amfani ta amfani da nau'i-nau'i inda amintaccen, amsa mai taimako ke 'zaɓa' akan mai matsala.
Koyar da mataimakiyar coding don fifita daidaitattun bayanai, ingantaccen rubuce-rubuce akan masu buggy ta amfani da kwatancen masu haɓakawa.
Salon taƙaitawa don haka samfura sun fi son taƙaitaccen taƙaitaccen bayani, amintaccen taƙaitaccen bayani akan na magana ko na hallucined
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Direct Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Haɓaka fifikon fifikon ƙima
Tambayoyin da ake yawan yi
What is Direct Preference Optimization?
Haɓaka fifiko kai tsaye (DPO) hanya ce ta daidaita ƙirar harshe tare da abubuwan son ɗan adam ba tare da horar da keɓantaccen samfurin lada ko gudanar da koyo na ƙarfafawa ba. Ya ruguje wani hadadden bututun mai matakai da yawa zuwa hasarar horo guda daya mai tsayayye.
Menene DPO ya kawar idan aka kwatanta da RLHF na gargajiya?
Babban fa'idar DPO shine cire ƙayyadadden samfurin lada da madaukin samfurin RL, inganta manufofin kai tsaye akan nau'ikan zaɓin zaɓi maimakon.
Wadanne bayanai ne misalin horon DPO guda ya kunsa?
DPO yana horar da nau'i-nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i nau'i)) na DPO yana horar da su: sauri da ɗaya da aka fi so ('zaɓi') da ɗayan da ba a so ('an ƙi') amsa.
Wace rawa samfurin tunani ke takawa a DPO?
Maganar daskararre (yawanci samfurin SFT) yana ɗaukar asarar; ma'aunin beta yana sarrafa yadda manufar horarwa zata iya motsawa daga gare ta.
A DPO, ina ake wakilta 'lada'?
Samuwar DPO yana nuna lada a fakaice a cikin rabon yuwuwar log tsakanin manufofi da tunani, don haka ba a buƙatar takamaiman samfurin lada.
Menene ma'aunin beta (zazzabi) a cikin DPO ke sarrafa?
Beta yana sarrafa ƙayyadaddun ƙayyadaddun KL: mafi girman beta yana kiyaye manufofin kusa da abin da ake tunani, ƙananan beta yana ba da ƙarin karkacewa.