Tokenisation
Tokenisation nintambwe ikata inyandiko mubice bito bita ibimenyetso, ibice urugero rwururimi rusoma kandi ruhanura.
Incamake
It quietly shapes cost, context limits, and even how well a model handles spelling and rare words.
Kwibira cyane
Mbere yuko icyitegererezo kibona inyandiko yawe, tokenizer ayigabanyamo ibimenyetso, ubusanzwe ni uduce duto duto kuruta amagambo yose cyangwa inyuguti imwe. Ijambo 'kutishima' rishobora guhinduka 'un', 'umunezero', cyangwa 'tokenisation' rishobora kwigabanyamo 'ikimenyetso' na 'ization'. Amagambo asanzwe akunze gushushanya kumurongo umwe, mugihe amagambo adasanzwe, amazina, cyangwa code igabanijwemo byinshi. Buri kimenyetso noneho gishushanyirizwa nimero y'indangamuntu icyitegererezo gihinduka muri vector. Ibi bifite akamaro mubyukuri kubera ko moderi ifite imiterere ihamye ya Windows yapimwe mubimenyetso, hamwe na fagitire ya APIs kuri buri kimenyetso, bityo itegeko rikomeye ryicyongereza ryintoki ni inyuguti 4 cyangwa amagambo 0,75 kuri buri kimenyetso. Tokenisation isobanura kandi icyitegererezo cyicyitegererezo: kubara inyuguti cyangwa gukora imyandikire nyayo biragoye kuko icyitegererezo kibona uduce, ntabwo ari inyuguti kugiti cye.
Ubushishozi
LLMs nyinshi zigezweho zikoresha subword tokenisation nka Byte Pair Encoding (BPE) cyangwa byte-urwego rwabyo. BPE itangirira ku nyuguti kandi igahuza inshuro nyinshi yegeranye kugirango yubake amagambo ahamye (akenshi 30.000 kugeza 100.000+ ibimenyetso). Iringaniza ibintu bibiri bikabije: ijambo-urwego tokenisation ntishobora gukoresha amagambo atagaragara, mugihe inyuguti-urwego ikora urutonde rurerure cyane. Subwords reka icyitegererezo kigereranya umurongo uwo ariwo wose, harimo amakosa yamagambo namagambo mashya, muguhimba ibice bizwi, mugihe bikurikiranye bigufi.
Ingaruka z'Ingamba
Ibyemezo bisobanutse
Iragufasha gutandukanya ibyifuzo bya tekiniki bisobanutse nururimi rwo kwamamaza.
Igiciro na bije
Urashobora kubaza ibibazo byiza byo gushyira mubikorwa mbere yo gukoresha amafaranga cyangwa igihe.
Itsinda hamwe nakazi
Amakipe asangiye ibitekerezo akora ibicuruzwa byiza, politiki, nibyemezo byo kwiga.
Ejo hazaza
Tokenisation nigice cyubushakashatsi gikora neza kuko kigabanya imikorere nuburinganire. Indimi zigaragaza ibice byinshi zigura amafaranga menshi kandi zigakoresha imiterere byihuse, bityo ubutabera mu ndimi nyinshi ni impungenge zifatika zikemurwa nijambo ryiza, ryuzuye. Abashakashatsi kandi barimo gukora ubushakashatsi kuri moderi idafite ibimenyetso cyangwa byte-byte (nka ByT5) kandi bize tokenisation ishobora gukuraho intambwe yoroheje yahinduwe intoki. Kuri ubu, tegereza amagambo manini, ubwenge bwindimi nyinshi zikoresha indimi nyinshi, hamwe no kwiyongera kubakoresha kumenya ibiciro bishingiye ku bimenyetso hamwe ningengo yimari.
Gushyira mu bikorwa Isi
Igiciro cya API kubitegererezo nka GPT na Claude byishyurwa kuri buri cyinjijwe n'ibisohoka, bityo ibimenyetso byerekana ibimenyetso bigira ingaruka kubiciro.
Imipaka-idirishya ntarengwa (urugero, 128K cyangwa 200K ibimenyetso) bipimirwa mubimenyetso, ukerekana umubare winyandiko cyangwa code ushobora gushiramo.
Abashinzwe iterambere bakoresha ibimenyetso (nka tiktoken) kugirango bagereranye ingano yihuse hamwe nibirimo mbere yo kohereza ibyifuzo.
Tokenisation isobanura impamvu abanyamideli barwanira kubara inyuguti mwijambo cyangwa guhindura umugozi, kubera ko babonye uduce duto duto, ntabwo ari inyuguti.
Ingaruka & Kurinda
Amakipe atandukanye arashobora gukoresha ijambo rimwe muburyo butandukanye, sobanura intera hakiri kare.
Ibipimo birashobora kugaragara bikomeye mugihe imikorere-yisi-itaringaniye.
Kwirengagiza ubuziranenge bwamakuru na gahunda yo gusuzuma akenshi bitanga ibisubizo byoroshye.
Igishushanyo mbonera
Tangira nururimi rusobanutse rwibisubizo ukeneye.
Toranya intsinzi imwe hamwe nuburyo bumwe bwo gutsindwa mbere yo kwipimisha.
Koresha umuderevu muto hamwe namakuru ahagarariye, ntabwo ari demo yashizweho.
Inyandiko aho Tokenisation ifasha nuburyo bworoshye bworoshye.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Interuro Igice
Ibibazo bikunze kubazwa
What is Tokenization?
Tokenisation nintambwe ikata inyandiko mubice bito bita ibimenyetso, ibice urugero rwururimi rusoma kandi ruhanura. Ituje ituje igiciro, imipaka igarukira, ndetse nuburyo icyitegererezo gikora neza imyandikire namagambo adasanzwe.
Niki 'ikimenyetso' murwego rwicyitegererezo cyururimi?
Tokens nibice byerekana icyitegererezo, mubisanzwe uduce duto duto, rimwe na rimwe amagambo yose cyangwa inyuguti imwe.
Ni ubuhe buryo bwo kwerekana ibimenyetso LLM igezweho ikoresha?
Uburyo bwa Subword nka BPE guhuza kenshi inyuguti ebyiri, kuringaniza intege nke zijambo ryiza-urwego hamwe nimiterere-urwego.
Kuki tokenisation ikora imirimo nko kubara inyuguti mwijambo rikomeye kuri LLMs?
Kuberako inyandiko zishyizwe mubimenyetso byerekana ibimenyetso, icyitegererezo nticyumva neza buri nyuguti, bigatuma inyuguti-urwego rwimirimo ikosa.
Kuki tokenisation ifite akamaro kubiciro bya API n'imbibi?
Abatanga fagitire kuri token hamwe nibisobanuro windows ifite ubunini mubimenyetso, kubara ibimenyetso byerekana gutwara ibiciro ndetse nuburyo ushobora gushiramo.
Kuki interuro imwe ishobora kugura ibimenyetso byinshi mundimi zimwe kuruta icyongereza?
Vocabularies yatojwe ahanini ku Cyongereza igabanya izindi ndimi mu bimenyetso byinshi, kuzamura ibiciro no gukoresha imvugo byihuse.