Nkwalite amụma nso
Proximal Policy Optimization (PPO) bụ nkwado mmụta mmụta algọridim kacha jikọtara ya na ezigbo nhazi ụdị asụsụ site na nzaghachi mmadụ.
Nchịkọta
It improves a policy in careful, small steps to avoid the instability that plagues naive policy gradient methods.
Ime miri emi
OpenAI webatara PPO na 2017 wee bụrụ onye na-arụ ọrụ n'azụ RLHF maka sistemụ dịka InstructGPT na ChatGPT. Isi ihe ịma aka na amụma-gradient RL bụ na otu mmelite buru ibu nwere ike daa arụmọrụ. PPO na-ekwu maka nke a na 'ebumnobi dochie anya': ọ na-atụ ole (ma ọ bụ obere) ọ ga-abụ na ihe omume abanyela na amụma ochie, na-amụba ọnụ ọgụgụ ahụ site na uru (lee otú omume ahụ si dị mma karịa ka a tụrụ anya ya), wee kewaa nha ahụ na obere nso dị ka 0.8 ruo 1.2. Nke a na-emetụta oke amụma nwere ike ịga n'ihu kwa mmelite, na-eme ka mmụta kwụsie ike ma ka na-ekwe ka nkwalite na-aga n'ihu. N'ụdị asụsụ RLHF, 'omume' na-emepụta akara ma ọ bụ nzaghachi, ụgwọ ọrụ na-abịa site na ụdị ụgwọ ọrụ, na ntaramahụhụ KL-iche na-eme ka ihe nlereanya ahụ ghara ịkpafu n'ebe dị anya na omume mbụ ya.
Nghọta nka nka
PPO na-ebuli ebumnobi ebibiri: min (ratio * uru, clip (ratio, 1-eps, 1+eps) * uru), ebe oke bụ ihe puru omume ime ihe karịrị ochie. A na-ejikarị enyocha uru ọhaneze na netwọ uru amụtara (nke nkatọ). Na RLHF, ngụkọta ụgwọ ọrụ na-ejikọta akara nrịbama-ụgwọ ọrụ yana ntaramahụhụ KL nke ọ bụla megidere amụma ntụnye aka, na-edozi uru ụgwọ ọrụ megide ịnọ nso na ụdị mbụ.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke nkwalite amụma nso
PPO ka na-esi ike mana ọ na-ewu ewu: ọ chọrọ netwọk bara uru dị iche, nlegharị anya hyperparameter kpachara anya, yana ọtụtụ mgbakọ. Nhọrọ ndị ọzọ dị mfe na-enweta ala, gụnyere DPO (enweghị RL ma ọlị) na GRPO, nke na-agbada netwọk uru site n'ịtụle uru sitere na otu nzaghachi egosipụtara ma kwadoro ụdị echiche na nso nso a. PPO ga-adịgide ebe nyocha nke amụma na-enyere aka n'ezie, mana mpaghara ahụ na-azụ ahịa ụfọdụ mgbagwoju anya maka ụzọ dị ọnụ ala.
Mmejuputa n'ezie n'ụwa
Ntuziaka nhazi nke ọmaGPT na ChatGPT iji soro ntuziaka na mmasị mmadụ site na RLHF
Ndị na-ahụ maka njikwa egwuregwu egwuregwu na-enye ọzụzụ, ngalaba izizi PPO tupu ụdị asụsụ
Mbelata nsị ma ọ bụ imeziwanye enyemaka site n'ịkwalite akara ngosi-ụgwọ ọrụ n'okpuru mmachi KL
Na-akwalite iji ngwá ọrụ ma ọ bụ omume ọtụtụ nzọụkwụ ebe a na-akwụ ụgwọ ihe nlereanya maka ịrụcha ọrụ nke ọma
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Proximal Policy Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nkwalite amụma ikwu otu
Ajụjụ a na-ajụkarị
What is Proximal Policy Optimization?
Proximal Policy Optimization (PPO) bụ nkwado mmụta mmụta algọridim kacha jikọtara ya na ezigbo nhazi ụdị asụsụ site na nzaghachi mmadụ. Ọ na-eme ka amụma na-akpachapụ anya, obere usoro iji zere enweghị ntụkwasị obi nke na-emebi ụzọ mmụgharị nke amụma enweghị isi.
Kedu nsogbu 'nchịkọta' PPO na-ekwu isi?
Ichichi oke nke puru omume na-egbochi mmelite ọ bụla ịkwaga amụma ahụ tere aka, nke bụ isi mmalite nke enweghị ntụkwasị obi na ụzọ amụma-gradient.
N'ụdị asụsụ RLHF nwere PPO, kedụ ebe mgbama ụgwọ ọrụ na-abịakarị?
Ụdị ụgwọ ọrụ na-enye ụgwọ ọrụ scalar maka nzaghachi ewepụtara, ebe ọ bụ na mmadụ nweta akara ọ bụla n'oge RL agaghị ekwe omume.
Kedu ihe ntaramahụhụ KL-divergence na-eme na RLHF dabere na PPO?
Ntaramahụhụ KL nke otu-token megidere amụma izizi (ntụaka) na-akụda ihe nlereanya ahụ ịgbanwe omume ya nke ukwuu ma na-achụ ụgwọ ọrụ.
Kedu ọrụ nke netwọk uru (onye nkatọ) na PPO?
PPO bụ usoro onye na-eme ihe nkiri; netwọk bara uru na-atụle ụgwọ ọrụ a na-atụ anya ya, na-eme atụmatụ atụmatụ uru (mgbe site na GAE) nke na-ebelata ọdịiche.
Kedu ihe 'uru' na-anọchi anya na PPO?
Uru ahụ na-atụ etu ihe omume ahọpụtara si ka nke ọma (ma ọ bụ nke ka njọ) dabere na atụmatụ uru, na-agwa amụma omume ndị a ga-emesi ike.