Overview
Kufanotaura kunotevera-chiratidzo ndicho chinangwa chinonyengera chiri kuseri kweGPT-maitiro emhando: kupihwa zvese kusvika zvino, fungidzira chikamu chinotevera chemavara. Inodzokororwa mabhiriyoni enguva, basa rimwe chete iri rinogadzira mhando dzinonyora, kufunga, uye kutaura.
Next-Token Prediction chikamu chemutauro-AI stack inoshandiswa kuverenga, kugadzira, kuronga, uye kushandura zvinyorwa uye kutaura pamwero.
Deep Dive
Kufanotaura kunotevera-chiratidzo kunodzidzisa modhi kugovera zvingangoitika kune inotevera tokeni yakapihwa ese matokeni apfuura. Mavara anotanga kuputswa kuita tokeni (subword zvidimbu) netokenizer senge byte-pair encoding. Decoder-chete Transformer inoverenga kutevedzana kubva kuruboshwe kuenda kurudyi uye inoburitsa mukana wekugovera pamusoro pemazwi ese echinzvimbo chinotevera. Panguva yekudzidziswa, modhi inoratidzwa yakakura mameseji corpora uye inorangwa pese painopa yakaderera mukana kune chaiyo inotevera tokeni. Panguva yechizvarwa, iyo modhi sampuli kana nemakaro inotora chiratidzo, inoiwedzera, uye inodzokorora iyi loop autoregressively. Ichi chinangwa chimwe chinoyera zvinoshamisa: GPT-2, GPT-3, uye vatevedzeri vese vakadzidza girama, chokwadi, dudziro, uye kufunga nekuita zvakanaka kwazvo kufanotaura chinotevera chiratidzo.
Technical Insight
Iyo kiyi nzira ndeye causal (masked) kuzvidzora: kana uchifanotaura chinzvimbo N, modhi inogona kungoenda kune zvinzvimbo 1 kuburikidza neN-1, kwete ramangwana. Iyo yekubuda layer ino gadzira iyo yekupedzisira yakavanzika mamiriro pane yemazwi uye inoshandisa softmax kuwana zvingangoitika. Kudzidzira kunoderedza mhiri-entropy, zvakaenzana nekuwedzera mukana wezvinyorwa zvinoonekwa. Sampling inodzora senge tembiricha uye yepamusoro-p reshape iyo kugovera pakunongedza kutengeserana kunze kwekusika kunopesana nekuvimbika.
Kubata Next-Token Prediction
Kufanotaura kunotevera-chiratidzo ndicho chinangwa chinonyengera chiri kuseri kweGPT-maitiro emhando: kupihwa zvese kusvika zvino, fungidzira chikamu chinotevera chemavara. Inodzokororwa mabhiriyoni enguva, basa rimwe chete iri rinogadzira mhando dzinonyora, kufunga, uye kutaura. Next-Token Prediction chikamu chemutauro-AI stack inoshandiswa kuverenga, kugadzira, kuronga, uye kushandura zvinyorwa uye kutaura pamwero. Kuti uvake kunzwisisa kwakadzama, tora Next-Token Prediction semuenzaniso wekushandisa, kwete chinhu chimwe chete: tsanangura zvinodikanwa, jekesa fungidziro, uye patsanura izvo zvinogona kuitwa nehurongwa hwakavimbika kubva kune zvichiri kuda kutonga kwenyanzvi.
Mukuita, zvikwata zvakasimba zvinoshandisa Next-Token Prediction dhizaini zvinokurudzira, kudzoreredza, uye kuongorora zvishwe seimwe yakabatanidzwa yekutaurirana system. Ivo vanonyora zvakajeka maitiro ebudiriro, bvunzo vachipokana ne data rechokwadi uye mafambiro ebasa, uye iterate zvichibva pane zvakacherechedzwa maitiro ekutadza kwete kuhwina-nguva imwe chete yebhenji. Apa ndipo apo kunzwisisa kwe theoretical kunoshanduka kuve kugona kwakasimba pane chigadzirwa, mutemo, uye mashandiro.
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana. Panguva imwecheteyo, chokwadi cheHallucified chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana kutsvagisa zvinobuda. Nzira yakatsiga ndeyekubatanidza kukurumidza kuyedza nekutonga: mhanyisa vatyairi vendege, tora humbowo, buritsa matanda esarudzo, uye urambe uchivandudza chengetedzo semaitiro emuenzaniso, zvinotarisirwa nemushandisi, uye zvinodikanwa zvekutonga.
Strategic Impact
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana.
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana.
Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora.
Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Real-World Implementation
Inosimbisa ChatGPT nevabatsiri vakafanana kugadzira mhinduro dzenhaurirano tokeni imwe panguva.
Autocomplete uye kodhi mazano mune zvishandiso seGitHub Copilot sezvaunonyora.
Kunyora maemail, zvinyorwa, uye kopi yekushambadzira kubva kupfupi kukurumidza.
Chaiyo-nguva mameseji chizvarwa mukunyora vabatsiri vanopedzisa mitsara yako.
Maitiro Ekuita
Next-Token Prediction mukuita
Inosimbisa ChatGPT nevabatsiri vakafanana kugadzira mhinduro dzenhaurirano tokeni imwe panguva.
Kusimbaradza ChatGPT uye vabatsiri vakafanana kuti vaunze mhinduro dzekukurukurirana tokeni imwe panguva Matimu anowanzo kuwana mibairo iri nani kana achinge atsanangura mabhindauko emhando yepamusoro, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
Next-Token Prediction mukuita
Autocomplete uye kodhi mazano mune zvishandiso seGitHub Copilot sezvaunonyora.
Autocomplete uye kodhi mazano mumidziyo yakaita seGitHub Copilot paunenge uchinyora Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye tarisa zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
Next-Token Prediction mukuita
Kunyora maemail, zvinyorwa, uye kopi yekushambadzira kubva kupfupi kukurumidza.
Kudhirowa maemail, zvinyorwa, uye kopi yekushambadzira kubva kupfupi kukurumidza Matimu anowanzo kuwana zvirinani mhedzisiro kana vachitsanangudza emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
Next-Token Prediction mukuita
Chaiyo-nguva mameseji chizvarwa mukunyora vabatsiri vanopedzisa mitsara yako.
Chaiyo-nguva mameseji kugadzirwa mukunyora vabatsiri vanopedza mitsara yako Matimu anowanzo kuwana zvirinani mhedzisiro kana vachitsanangudza zvemhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye tarisa zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
Njodzi & Guardrails
Chokwadi chehuroyi chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana tsvakiridzo.
Kunzwa nekukasira kunogona kugadzira mhedzisiro isingaenderane pane zvikumbiro zvakafanana.
Sensitive text data inogona kuburitswa kana zvidhiraivho zvisina kusimba.
Implementation Roadmap
Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa.
Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Mhinduro dzepasi neakavimbika masosi pese pazvine basa.
Mhinduro dzepasi neakavimbika masosi pese pazvine basa. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda.
Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva.
Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.