Tokenisation na Byte Pair Encoding
Tokenisation igabanya inyandiko mubice bito urugero rwururimi rusoma, kandi Byte Pair Encoding (BPE) nuburyo buzwi bwo kubaka ayo magambo.
Incamake
It balances having a manageable vocabulary against handling any word the model might encounter.
Kwibira cyane
Ururimi rwindimi ntirubona inyuguti mbisi cyangwa amagambo yose - babona ibimenyetso, indangamuntu yuzuye yashizwe mubice byinyandiko. Guhitamo ibyo bice ni ubucuruzi: ijambo-urwego rwamagambo ni manini kandi aruniga kumagambo atagaragara cyangwa yanditse nabi, mugihe urwego-rwerekana urwego rukurikirana. Byte Pair Encoding ikubita hagati. Yatijwe kuva 1990-data-compression algorithm, BPE itangirira kumuntu kugiti cye (cyangwa byite mbisi) kandi igahuza inshuro nyinshi guhuza ibice byinshi mubimenyetso bishya, bikura amagambo yerekeza kumagambo asanzwe. Amagambo akunze kuba ikimenyetso kimwe, mugihe amagambo adasanzwe yacitsemo ibice bikoreshwa. Byte-urwego BPE, ikoreshwa na moderi ya GPT, ikora kuri bytes mbisi kuburyo ishobora kwerekana inyandiko iyo ari yo yose ya Unicode - harimo emoji n'ururimi urwo arirwo rwose - nta kunanirwa-amagambo.
Ubushishozi
BPE imyitozo irarikira kandi itwarwa ninshuro. Guhera ku nyuguti zifatizo, ibara ibimenyetso byegeranye byombi hejuru ya corpus kandi igahuza ibisanzwe, byandika buri guhuza nkitegeko. Gusubiramo ibi ibihumbi inshuro zitanga urutonde rwateganijwe hamwe namagambo ahamye. Mugusuzuma, inyandiko yashizwemo mugukurikiza ayo mategeko yo guhuza. Niyo mpanvu ibimenyetso bibarwa bidakunze guhuza ijambo kubara: umwanya, inyuguti nkuru, namagambo adasanzwe byose bihindura uburyo ibice byanditse mubimenyetso, kandi ijambo rimwe rishobora guhinduka ibimenyetso byinshi.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Kazoza ka Tokenisation na Byte Pair Encoding
Tokenisation iri gutekereza cyane. Byte- na inyuguti-urwego rwicyitegererezo nka ByT5, hamwe no kugaragara kwubusa cyangwa 'byte-latent' imyubakire, igamije guta amagambo ahamye rwose kuburyo moderi ikora ibyinjijwe nururimi urwo arirwo rwose. Abashakashatsi kandi barimo guhangana n’uburinganire bwerekana ibimenyetso - indimi nyinshi zitari Icyongereza n’umutungo muke kuri ubu zigura ibimenyetso byinshi cyane kuri buri nteruro, kuzamura ibiciro no kugabanya imiterere ifatika. Tegereza ibimenyetso byerekana kode, imibare, nuburinganire bwindimi nyinshi, wongeyeho ubushakashatsi bwakomeje kugirango usubize imbibi ugana kuri byite mbisi.
Gushyira mu bikorwa Isi
Moderi ya GPT na Llama ikoresha BPE-yuburyo bwa tokenizers kugirango ihindure ibisobanuro mubiranga ibimenyetso byerekana urusobe.
Ibiciro bya API hamwe nimbonerahamwe-idirishya ntarengwa bipimirwa mubimenyetso, bityo tokenisation igira ingaruka itaziguye nigiciro hamwe ninyandiko ihuye.
Gukoresha emoji, kode, namagambo adasanzwe ubigabanyijemo ibice byongeye gukoreshwa cyangwa ibice byte.
Gushyigikira indimi nyinshi murugero rumwe rudafite inkoranyamagambo itandukanye kururimi, ukoresheje byte-urwego rwa kodegisi.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Tokenization and Byte Pair Encoding quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Byte-Byombi Encoding
Ibibazo bikunze kubazwa
What is Tokenization and Byte Pair Encoding?
Tokenisation igabanya inyandiko mubice bito urugero rwururimi rusoma, kandi Byte Pair Encoding (BPE) nuburyo buzwi bwo kubaka ayo magambo. Iringaniza kugira amagambo acungwa kurwanya ijambo iryo ari ryo ryose icyitegererezo gishobora guhura nacyo.
Niki tokenisation itanga urugero rwururimi rwo gusoma?
Icyitegererezo gikora ku kimenyetso - indangamuntu yuzuye ihagaze ku nyuguti, ijambo, cyangwa amagambo - kuruta imirongo mbisi.
Nigute Byte Pair Encoding yubaka amagambo yayo?
BPE itangirira ku nyuguti cyangwa bytes kandi umururumba uhuza inshuro ebyiri zegeranye, buhoro buhoro ugakora ijambo ryibisobanuro rusange.
Ni ikihe kibazo uburyo bwa subword uburyo nka BPE bukemura ugereranije nijambo-urwego tokenisation?
Ijambo-urwego rwamagambo ananirwa kumagambo atagaragara; subword tokenisation isenya amagambo adasanzwe mubice bizwi, wirinda ibibazo bitari amagambo mugihe ugumya amagambo gucungwa.
Ni izihe nyungu za byte-urwego BPE, nkuko bikoreshwa muri moderi ya GPT?
Gukorera kuri bytes mbisi bivuze ko ibishoboka byose byinjizwa bishobora gushyirwaho kodegisi, kubwibyo rero nta nyuguti zitazwi rwose utitaye ku rurimi cyangwa ikimenyetso.
Ni ukubera iki icyitegererezo cyerekana ibimenyetso akenshi gitandukanya numubare wamagambo mu nteruro?
Subword tokenisation irashobora kumena ijambo rimwe mubice byinshi kandi byunvikana kumwanya hamwe nurubanza, bityo ibimenyetso bibarwa gake gake kubara ijambo.