Ingantaccen Manufofin Kusa
Manufa Proximal Proximal (PPO) shine ingantaccen koyo algorithm wanda ya fi alaƙa da ingantaccen tsarin harshe daga ra'ayin ɗan adam.
Dubawa
It improves a policy in careful, small steps to avoid the instability that plagues naive policy gradient methods.
Zurfafa nutsewa
OpenAI ne ya gabatar da PPO a cikin 2017 kuma ya zama dokin aiki a bayan RLHF don tsarin kamar InstructGPT da ChatGPT. Babban ƙalubalen a cikin manufofin-gradient RL shine cewa ɗaukakawa mai girma fiye da kima na iya rushe aikin. PPO yayi magana da wannan tare da 'maƙasudin maye gurbin': yana auna nawa (ko žasa) mai yiwuwa wani aiki ya zama daidai da tsohuwar manufofin, yana ninka wannan rabo ta fa'ida (nawa aikin ya fi yadda ake tsammani), kuma yana tsara rabo zuwa ƙaramin kewayo kamar 0.8 zuwa 1.2. Wannan yana ɗaukar nisa yadda manufofin za su iya motsawa kowane sabuntawa, kiyaye koyo da kwanciyar hankali yayin da har yanzu ke ba da damar ci gaba mai ƙarfi. A cikin samfurin harshe na RLHF, 'aikin' yana haifar da alama ko amsawa, ladan yana fitowa daga samfurin lada, kuma hukuncin rarrabuwar kawuna na KL yana kiyaye ƙirar daga yin nisa sosai daga ainihin halayensa.
Fahimtar Fasaha
PPO yana haɓaka haƙiƙa da aka yanke: min (rabo * fa'ida, shirin shirin (rabo, 1-eps, 1+eps) * fa'ida), inda rabo shine yuwuwar sabon-sama-tsohon aiki. Yawancin fa'idodi ana ƙididdige su tare da Ƙimar Fa'ida Gabaɗaya da cibiyar sadarwar ƙima (mai suka). A cikin RLHF, jimlar lada ta haɗu da ƙima-samfurin lada tare da hukuncin kowane-alama ta KL akan manufar tunani, daidaita ribar lada akan kasancewa kusa da ƙirar asali.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar Inganta Manufofin Kusa
PPO ya kasance mai ƙarfi amma sanannen amana ne: yana buƙatar cibiyar sadarwar ƙima daban, mai da hankali ta hyperparameter, da ƙididdige yawa. Zaɓuɓɓuka masu sauƙi suna samun ƙasa, gami da DPO (babu RL kwata-kwata) da GRPO, wanda ke sauke hanyar sadarwar ƙimar ta kimanta fa'idodi daga ƙungiyoyin amsoshi da aka ƙididdigewa kuma ya ba da ikon ƙirar tunani na kwanan nan. PPO za ta dage inda bincike kan manufofin ke taimakawa da gaske, amma filin yana yin ciniki da wasu hadaddun sa don hanyoyi masu rahusa.
Aiwatar da Gaskiyar Duniya
Ingantattun InstructGPT da ChatGPT don bin umarni da abubuwan da mutane ke so ta hanyar RLHF
Horar da wasan-wasa da masu sarrafa mutum-mutumi, asalin yankin PPO kafin ƙirar harshe
Rage yawan guba ko haɓaka taimako ta hanyar haɓaka ƙimar samfurin lada a ƙarƙashin ƙuntatawar KL
Haɓaka amfani da kayan aiki ko ɗabi'ar wakili mai matakai da yawa inda aka sami lada ga ƙira don kammala ayyuka daidai
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Proximal Policy Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Inganta Manufofin Dangi na Ƙungiya
Tambayoyin da ake yawan yi
What is Proximal Policy Optimization?
Manufa Proximal Proximal (PPO) shine ingantaccen koyo algorithm wanda ya fi alaƙa da ingantaccen tsarin harshe daga ra'ayin ɗan adam. Yana inganta manufa cikin hankali, ƙananan matakai don guje wa rashin zaman lafiya da ke addabar hanyoyin karkatar da manufofin butulci.
Wace matsala ce 'yanke' PPO ke magance da farko?
Yanke rabon yuwuwar yana hana duk wani sabuntawa guda ɗaya daga matsar da manufofin da nisa, wanda shine babban tushen rashin zaman lafiya a cikin hanyoyin haɓakar manufofin.
A cikin samfurin-harshen RLHF tare da PPO, daga ina siginar lada yawanci ke fitowa?
Samfurin lada yana ba da lada mai ƙima don amsawar da aka samar, tunda samun ɗan adam maki kowane fitarwa yayin RL ba zai yuwu ba.
Menene hukuncin bambance-bambancen KL ke yi a cikin tushen PPO RLHF?
Hukuncin KL na kowane-alamu akan ainihin manufar (bayani) yana hana ƙirar canza halayensa sosai yayin neman lada.
Menene rawar cibiyar sadarwar ƙima (mai sukar) a cikin PPO?
PPO hanya ce ta masu sukar wasan kwaikwayo; cibiyar sadarwar ƙimar tana ƙididdige lada da ake tsammanin, yana ba da damar kimanta fa'ida (sau da yawa ta hanyar GAE) wanda ke rage bambance-bambance.
Menene 'fa'idar' ke wakilta a cikin PPO?
Amfanin yana auna yadda mafi kyau (ko mafi muni) aikin da aka zaɓa ya danganta da ƙimar ƙimar, yana gaya wa manufofin ayyukan da za a ƙarfafa.