Token bi ci topp
Next-token prediction mooy mébet bu yomb bi ci ginaaw xeetu GPT: jox lépp ba leegi, xalaatal mbind mi ci topp.
Résumé
Repeated billions of times, this single task produces models that write, reason, and converse.
Plongeur bu xóot
Xalaatal token bi ci topp dafay tàggat ab model ngir jox probabilite token bi ci topp buñu jox token yi ko jiitu yépp. Bind bi dañu koy njëkka xaaj ay token (piyeesu sous-baatu) jaaraleko ci tokenizer bu melni kodage byte-pair. Transformatër bu dekodeer kese mooy jàng limu toppalante ci cammoy ba ci ndeyjoor ba noppi génne ab séddaleb probabilite ci vocabulaire bi yépp ngir barab bi ci topp. Ci diiru tàggat-yaram, dañuy wane xeetu mbind mu bari te dañu koy yar saa yu joxe probabilite bu néew ci token bi ci topp. Ci jamonoy defar, model bi dafay jël misaal wala mu tànn benn token, yokk ko, ba noppi baamtu loop bi ci anam wu autoregressif. Benn mébet bii dafa am solo lool: GPT-2, GPT-3, ak ñi ci topp ñoom ñépp jàng nañu nàfar, dëgg, tekki làkk, ak xalaat ci xam luy waaja am ci token bi ci topp.
Gis-gis xarala
Mekanism bi gëna am solo mooy bàyyi xel ci sa bopp (maske) : sooy wax luy waaja am ci position N, model bi mën na bàyyi xel ci position 1 ba N-1, du musa bàyyi xel ci ëlëg. Couche biy génne dafay wane stade bu mujj bi nëbbu ci vocabulaire bi ba noppi jëfandikoo softmax ngir am probabilité yi. Taggat yaram dafay wàññi cross-entropy, mu méngoo ak gëna yokk wërsëgu mbind mi ñuy seetlu. Kontrolu sampling yu melni tàngoor ak top-p dañuy soppi jëmmu séddale bi ci inference ngir dindi sos lu bees ak wóor.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu Next-Token
Xalaatal token bu ci topp bi mungi and ak xeetu lakk yu magg yi ci jamono jii te dina des ci yax bu juntuwaay yu bees yi. Gëstu daf koy yokk ci palanteer yu gëna gudd, dekodaas speculatif ak paralel ngir gaawaay, ak mébetu wax luy waaja am ci token yu bari yuy xalaat token yu bari ci ëlëg benn yoon. Jàngat bu am doole ci feedback nit ñi ci kaw ngir méngale ay génne. Frontier bi mingi def benn mébet bu yomb bi gëna xéewale, gëna gaaw, gëna mëna yor ci escale bu gëna mag.
Doxal ci àdduna dëgg
Dafay may ChatGPT ak ay ndimbal yu mel noonu ngir ñu mëna defar ay tontu ci waxtaan benn token benn yoon.
Defal sa bopp ak digle kode ci jumtukaay yu melni GitHub Copilot sooy bind.
Defar ay imeel, ay article ak ay kopi fësal njaay ci diir bu gàtt.
Defar mbind ci jamono dëgg ci assistant bind yuy jeexal sa tegtal.
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Next-Token Prediction quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Taggat ci wax luy am ci token yu bari
Laaj yi ñuy faral di laaj
What is Next-Token Prediction?
Next-token prediction mooy mébet bu yomb bi ci ginaaw xeetu GPT: jox lépp ba leegi, xalaatal mbind mi ci topp. Buñu ko baamtu ay miliyaar ciy yoon, benn liggéey boobu dafay defar ay misaal yuy bind, xalaat ak wëlbatiku.
Lan la modelu wax luy am ci token bi ci topp di génne ci jéego bu nekk?
Modèle bi dafay defar probabilite ngir bépp token bu ci topp, ginaaw ga ñu tànnee benn ci sampling wala tànneef bu bëgg xaalis.
Ban xeetu bàyyi xel la dekodeer bu nuroo ak GPT di jëfandikoo?
Maskeer causal dafay fexe ba position N mënul gis position yi ko jiitu rek, ba noppi baña yàq tabb prediction bu cammoy-ci-ndijoor.
Luy tekki ci jamonoy 'autorégressif' fii?
Generation autorégressive dafay defar benn token, yokk ko ci contexte bi, ba noppi baamtu loop bi.
Ban fonction perte lañuy gëna wàññi ci diiru tàggat yaram?
Entropie croix dafay yar xeetu joxe probabilite bu néew ci token bi ci topp dëgg, mu méngoo ak probabilite bu gëna mag.
Luy yokk tàngoor wi ci diiru jël misaal?
Tàngoor wu gëna rëy dafay dalal seddaleb probabilite yi, di yokk aleatoire bi; tàngoor wu wàññeeku dafay tax tànneef yi gëna am njariñ.