Gukwirakwiza Politiki Yegeranye
Politiki ya Optimisiyonike (PPO) ni imbaraga zo kwiga algorithm zifitanye isano cyane no guhuza neza imvugo y'ururimi ruva mubitekerezo byabantu.
Incamake
It improves a policy in careful, small steps to avoid the instability that plagues naive policy gradient methods.
Kwibira cyane
PPO yatangijwe na OpenAI muri 2017 ihinduka inzu yakazi inyuma ya RLHF kuri sisitemu nka InstructGPT na ChatGPT. Ikibazo cyibanze muri politiki-icyiciro cya RL ni uko ivugurura rimwe rirenze rishobora gusenya imikorere. PPO ikemura iki kibazo hamwe na 'clips surrogate objectif': ipima uburyo burenze (cyangwa buto) bishoboka ko igikorwa cyahindutse na politiki ishaje, kugwiza icyo kigereranyo kubwinyungu (uko ibikorwa byari byiza kuruta uko byari byitezwe), kandi bigabanya ikigereranyo ku ntera ntoya nka 0.8 kugeza 1.2. Ibi byerekana intera politiki ishobora kwimuka kuri buri gihe, igakomeza kwiga neza mugihe ikomeje kwemerera gutera imbere. Mu ndimi-moderi RLHF, 'igikorwa' gitanga ikimenyetso cyangwa igisubizo, ibihembo biva muburyo bwo guhemba, kandi igihano cya KL-gutandukana bituma icyitegererezo kidahinduka kure yimyitwarire yambere.
Ubushishozi
PPO yerekana intego zaciwe: min (igereranyo * inyungu, clip (igereranyo, 1-eps, 1 + eps) * inyungu), aho igipimo aricyo gishya-kirenze-ibikorwa byashoboka. Inyungu zisanzwe zigereranijwe hamwe na rusange muri rusange hamwe no kwiga agaciro (kunegura). Muri RLHF, ibihembo byose bihuza amanota-yicyitegererezo amanota hamwe na buri kimenyetso cya KL kuri politiki yerekana, kuringaniza inyungu zigihembo cyo kuguma hafi yicyitegererezo cyambere.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza ka Politiki Yegeranye
PPO ikomeza gukomera ariko izwi cyane: ikenera urusobe rwagaciro rwihariye, kwitondera hyperparameter witonze, hamwe na compte nyinshi. Ubundi buryo bworoshye burimo kwiyongera, harimo DPO (nta RL na gato) na GRPO, igabanya umuyoboro wagaciro mugereranya ibyiza biva mumatsinda yibisubizo byatanzwe kandi byatanze uburyo bwo gutekereza vuba. PPO izakomeza aho ubushakashatsi kuri politiki bufasha rwose, ariko umurima urimo ucuruza bimwe mubintu bigoye kuburyo buhendutse.
Gushyira mu bikorwa Isi
Kuringaniza neza AmabwirizaGPT na ChatGPT gukurikiza amabwiriza nibyifuzo byabantu ukoresheje RLHF
Kumenyereza gukina-gukina na robo yubugenzuzi, PPO yumwimerere mbere yicyitegererezo cyururimi
Kugabanya uburozi cyangwa kunoza ubufasha mugukoresha amanota-yicyitegererezo amanota ya KL
Kunoza ibikoresho-gukoresha cyangwa imyitwarire yintambwe nyinshi aho icyitegererezo gihembo cyo kurangiza imirimo neza
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Proximal Policy Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Gukwirakwiza Politiki Yerekeye Amatsinda
Ibibazo bikunze kubazwa
What is Proximal Policy Optimization?
Politiki ya Optimisiyonike (PPO) ni imbaraga zo kwiga algorithm zifitanye isano cyane no guhuza neza imvugo y'ururimi ruva mubitekerezo byabantu. Itezimbere politiki muburyo bwitondewe, buto kugirango wirinde ihungabana ryibasiye naive politike ya gradient.
Ni ikihe kibazo 'gukuramo' PPO gikemura cyane cyane?
Gukuraho igipimo gishoboka kibuza ivugurura iryo ariryo ryose kwimura politiki kure cyane, niyo soko nyamukuru yo guhungabana muburyo bwa politiki-buhoro buhoro.
Mundimi-moderi RLHF hamwe na PPO, ibimenyetso byigihembo mubisanzwe bituruka he?
Icyitegererezo cyigihembo gitanga ibihembo bya scalar kubisubizo byatanzwe, kubera ko abantu batanga amanota yose mugihe cya RL ntibyashoboka.
Niki gihano KL-divergence ikora muri RLHF ishingiye kuri PPO?
Igihano kuri KL kuri politiki yumwimerere (reference) ibuza icyitegererezo guhindura imyitwarire ikabije mugihe wiruka ibihembo.
Ni uruhe ruhare rw'urusobe rw'agaciro (kunegura) muri PPO?
PPO nuburyo bukinisha abakinnyi; agaciro urusobe rugereranya ibihembo biteganijwe, bigufasha kugereranya inyungu (akenshi binyuze muri GAE) bigabanya itandukaniro.
Niki 'akarusho' kagereranya muri PPO?
Inyungu ipima uburyo bwiza (cyangwa bubi) igikorwa cyatoranijwe cyagereranijwe nigereranya ryagaciro, ubwira politiki ibikorwa byo gushimangira.