Kodage ñaari byte
Kodage Byte-Pair (BPE) algorithm la bu lalu ci kompresioŋ, muy tabax vokabileer ci boole ñaari màndarga yi gëna bari luñu koy jëfandikoo.
Résumé
It is the tokenizer behind GPT models, balancing tiny vocabularies of characters against huge vocabularies of whole words.
Plongeur bu xóot
BPE dafay tàmbali ci jëfandikoo mbind mi ni ay araf yu benn-benn (wala ay byte yu ñor). Ginaaw loolu mu lim bépp peer symbol bu dendoo, boole peer bi gëna bari ci token bu bees, ba noppi baamtu ko ay junniy yoon. Bépp boole dañu koy bind ni sàrt. Arafu yiñ gëna xam lu ci melni 'th', 'ing', wala baat yu bari dañuy dem ba nekk ay token yu benn, waaye baat yu bariwul ñuy xaajaloo nekk ay pàcc yu ndaw. Bi ñuy tàmbali, xeetu kompresioŋ la woon ci 1994, Sennrich ak ñeneen ñoo ko méngale ak NLP. ci 2016 ngir tekki masin. GPT-2 ak GPT-4 dañuy jëfandikoo BPE bu tollu ci byte, muy dox ci byte UTF-8 suko defee bépp araf, emoji wala làkk mën nañu ko kode saa yu nekk te du am benn njuumte ci vocabulaire bi.
Gis-gis xarala
Taggat BPE defar limu sàrti boole yuñ raññe. Ngir tokenize mbind mu bees, algorithm bi daf koy xaaj ay byte/karakteer ba noppi jëfandikoo boole yi ci benn rang bu njëkk ba amul benn sàrt bu méngoo. BPE-niveau byte dafay garanti fallback: ba ci benn màndarga buñu gisul dafay seey ci byte yi ci bokk, kon vocabulaire bu 256 byte boole ci boole jàngat yi dañuy muur lépp lu amul token UNK.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu kodage Byte-Pair
BPE desna tokenizer workhorse, waaye fitnaal bi dafay gëna yokk ci modelu byte- wala niveau karakter yuy salte tokenization bu leer, moytu quirks yu melni xaaj bu jaxasoo ci kode, math, wala script yu dul Angale. Gëstu ci architecture yu amul token ak tokenizers yuñ jàng ngir saafara jafe-jafe yi BPE am. Ba leegi, gaawaayam ak njariñu kompresioŋ dafay tekki ni vocabulaire yu nuroo ak BPE dina ñu dooleel LLM yu bari yi ci diir bu gàtt.
Doxal ci àdduna dëgg
GPT-2 ak GPT-4 dañuy jëfandikoo BPE ci niveau byte suko defee bépp arafu Unicode wala emoji mën nañu ko kode te du am benn njuumte.
Sistemu tekkikatu masin yi dañuy jëfandikoo BPE ngir xaaj baat yu bariwul wala yuñ boole mu nekk ay baat yuñ mëna jëfandikoowaat, ñu bokk ko ci làkk yi.
Bibliothèque tokenizers bu Hugging Face dafay tàggat vokabileer BPE ngir domen yuñ personaalise bu melni mbindu biomédical wala mbindu yoon.
Modèlu kod yi dañuy tokenise ràññeekaay yi ak baatu-caabi yi ak BPE, boole motif yu bari yu melni 'def' wala '==' ci benn token.
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Byte-Pair Encoding quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Tokenisasioŋ ak kodage ñaari byte
Laaj yi ñuy faral di laaj
What is Byte-Pair Encoding?
Kodage Byte-Pair (BPE) algorithm la bu lalu ci kompresioŋ, muy tabax vokabileer ci boole ñaari màndarga yi gëna bari luñu koy jëfandikoo. Mooy tokenizer bi ci ginaaw xeetu GPT, di ekilibre vocabulaire yu ndaw yu araf yi ak vocabulaire yu mag yu baat yu mat.
Lan mooy liggéey bi BPE wara baamtu ngir tabax vocabulaire bi?
BPE dafay lim ñaari màndarga yi ci wetam ba noppi boole benn bi gëna bari ci benn token bu bees, baamtu ko ay junniy yoon.
Lan la BPE di def ci mbind mi suñu tàmbali tàggat?
BPE mingi tàmbali ci yunit yi gëna ndaw (araf wala byte yu ñor) ba noppi màgg token yu gëna mag ci boole.
Lan moo waral BPE niveau byte moytu njuumti yi nekk ci bitti vocabulaire (UNK)?
Ndax vocabulaire bu njëkk bi dafa amaale 256 byte yépp, ba ci màndarga yiñ gisul dañuy wàcci ci seen representation byte.
Fan la algorithm BPE bawoo balaa NLP diko jëfandikoo?
BPE dañu ko dugal ci atum 1994 muy pexem kompresioŋ done, ginaaw ga ñu jëfandikoo ko ngir tokenization ci atum 2016.
Sooy tokenize mbind mu bees, naka la BPE di jëfandikoo sàrt yi mu jàng?
Regle yiñ boole dañu leen di def, tokenization daf leen di jëfandikoo ci anam wu bëgg lu njëkk ba keroog amul beneen sàrt bu méngoo.