Ubuyobozi bwa tekiniki

Gushimangira Kwiga Biturutse kubitekerezo byabantu

Reinforcement learning from human feedback uses human judgments to construct a training signal for a model’s behavior.

Umva iki gitaboAI-byavuzwe · ~2 min

Kuriyi page2 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Ejo hazaza h'imbaraga zo Kwiga Biturutse kubitekerezo byabantu
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

It can improve agreement with collected preferences, but does not guarantee truthful, safe, or universally preferred answers.

Kwibira cyane

A pretrained language model predicts plausible text, but plausible is not the same as helpful, honest, or safe. A common RLHF pipeline addresses these goals in stages. First, supervised fine-tuning teaches the model to follow instructions using human-written example answers. Next, humans compare pairs of model responses to the same prompt and pick the better one; these comparisons train a separate reward model that scores any response. Finally, the language model is optimized with reinforcement learning to produce responses the reward model rates highly. A penalty regularizes deviation from a reference policy; it does not eliminate exploitation of reward-model errors. RLHF was central to making ChatGPT-style assistants usable.

Gushyira mu bikorwa Isi

Guhuza umufasha wibiganiro kugirango yange ibyifuzo byangiza kandi bitange ibisubizo byingirakamaro, byubatswe neza aho kuba inyandiko yemewe.

Gutondekanya ibice bibiri byincamake kubantu bakunda gutoza icyitegererezo cyandika incamake abantu basanga ari ingirakamaro.

Kugabanya umusaruro wuburozi cyangwa kubogama mugushimira ibisubizo abatanga amanota yabantu bubaha kandi bafite umutekano.

Gukoresha DPO kuri dataset yikundwa na vanze ibisubizo kugirango uhuze moderi ifunguye isoko idakoresheje PPO yuzuye.

Ejo hazaza h'imbaraga zo Kwiga Biturutse kubitekerezo byabantu

RLHF irimo kugenda neza kandi igice cyikora. DPO hamwe nuburyo bujyanye nuburyo bukurikira busimbuza umuyoboro uremereye wa PPO kumakipe menshi, kandi RLAIF ikoresha ibitekerezo byatanzwe na AI (nko muri AI Itegekonshinga) kugirango igabanye ibiciro. Ubushakashatsi burimo guhangana nigihembo cyibihembo, kubogama kwa annotator, hamwe ningorabahizi zo gusuzuma ibisubizo birebire cyangwa byinzobere, hamwe nubuhanga nko kugenzura inzira no kujya impaka. Witege guhuza guhuza ibitekerezo byabantu na AI, ibimenyetso byinshi byigihembo birenze igikumwe kimwe, hamwe no kugenzura neza uwatanze ibyo akunda nindangagaciro bashiraho.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Reinforcement Learning From Human Feedback quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is Reinforcement Learning From Human Feedback?

Reinforcement learning from human feedback uses human judgments to construct a training signal for a model’s behavior. It can improve agreement with collected preferences, but does not guarantee truthful, safe, or universally preferred answers.

Niyihe ntego nyamukuru ya RLHF kurugero rwururimi?

RLHF iyobora icyitegererezo kubisubizo abantu bakunda, bikarushaho gufasha, kuba inyangamugayo, n'umutekano aho kuba byemewe gusa.

Ni ubuhe buryo bwo guhemba muri RLHF bwiga gukora?

Icyitegererezo cyigihembo cyahuguwe kubigereranyo byabantu kugereranya amanota kugirango batange ibisubizo, bitanga ikimenyetso politiki nziza.

What does a KL-divergence penalty against a reference policy aim to regularize in an RLHF objective?

The penalty discourages large policy shifts. It is not proof that the reward model cannot be exploited or that generated answers are correct.

Nigute amakuru yatoranijwe yakusanyirijwe muburyo bwo guhemba?

Annotator batoranya igisubizo cyatoranijwe mugereranya, itoza icyitegererezo cyo guhemba binyuze mu gihombo cya Bradley-Terry.

Ni izihe nyungu DPO (Optimisation itaziguye) itanga hejuru ya RLHF ya kera?

DPO ikura intego iturutse kubyo ukunda, irinda icyitegererezo cyigihembo cyihariye hamwe nintambwe yo kwiga ishimangira.