Subword Tokenisation
Subword tokenisation igabanya inyandiko mubice bito kuruta amagambo ariko binini kuruta inyuguti, nka 'token' wongeyeho 'ization'.
Incamake
It is the standard way modern language models turn text into the discrete IDs they actually process, balancing vocabulary size against meaning.
Kwibira cyane
Amagambo ni menshi cyane kubara (amagambo yaba menshi kandi akabura amagambo adasanzwe), mugihe inyuguti imwe itwara ibisobanuro bike kandi ikora urutonde rurerure cyane. Subword tokenisation ni ubwumvikane: ikomeza amagambo kenshi yose ariko ikavuna amagambo adasanzwe cyangwa akomeye mubice bifite ireme. 'Ibyishimo' bishobora guhinduka 'un', 'happyi', 'ness'. Algorithm nini zirimo Encoding ya Byte-Pair (ikoreshwa na GPT), WordPiece (ikoreshwa na BERT), na Unigram / SentencePiece (ikoreshwa na T5 hamwe na moderi nyinshi zindimi nyinshi). Ubu buryo bukoresha amagambo atagaragara neza, agabana ibice mumagambo afitanye isano ('gukina', 'gukina', 'gukina'), kandi ashyigikira ururimi urwo arirwo rwose. Buri karita yerekana ikarita yerekana indangamuntu, kandi izo ndangamuntu nizo moderi yo gushiramo icyitegererezo ihinduka mubice.
Ubushishozi
Algorithms zitandukanye zihitamo ijambo ryibisobanuro bitandukanye: BPE ihuza inshuro ebyiri hasi-hejuru, WordPiece itoranya ihuza byinshi byongera corpus bishoboka, kandi Unigram itangirana nijambo rinini hamwe na prunes ibimenyetso bitababaza cyane. WordPiece iranga ijambo-imbere ibice hamwe na '##' ibanziriza, mugihe SentencePiece ifata imyanya nkikimenyetso kidasanzwe bityo ikora ku buryo butaziguye ku mwandiko mbisi utabanje gutandukana ku kibanza cyera, cyiza ku ndimi zitagira umwanya.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza ka Subword Tokenisation
Subword tokenisation izaguma yiganje kuko irihuta kandi yoroheje, ariko intege nke zayo, gutandukana kubiharuro, kode, hamwe ninyandiko zidasanzwe, wongeyeho ibiciro byerekana ibimenyetso bitaringaniye mu ndimi, bitera ubushakashatsi muburyo bwa byte-na moderi idafite ibimenyetso. Witege ubwenge, birashoboka ko wize cyangwa uhuza ibimenyetso byerekana ibimenyetso hamwe nuburinganire bwiza bwindimi nyinshi kuburyo inyandiko itari icyongereza idahanwa nibimenyetso byinshi kuri buri nteruro.
Gushyira mu bikorwa Isi
BERT ikoresha WordPiece tokenisation, ikerekana ibice bikomeza nka '## ing' kugirango yubake amagambo yumwimerere.
T5 hamwe nicyitegererezo cyindimi nyinshi zikoresha SentencePiece, ikoresha indimi zitagira umwanya nkikiyapani.
Moderi yo kuganira yagabanije ijambo tekinike idasanzwe mubice bizwi aho kunanirwa kumagambo atazwi.
Tokenizers basangira ijambo ryibisobanuro kuri 'kwiruka', 'kwiruka', na 'kwiruka', kureka icyitegererezo rusange muri morphologie neza.
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Subword Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Interuro Igice
Ibibazo bikunze kubazwa
What is Subword Tokenization?
Subword tokenisation igabanya inyandiko mubice bito kuruta amagambo ariko binini kuruta inyuguti, nka 'token' wongeyeho 'ization'. Nuburyo busanzwe imiterere yindimi zigezweho zihindura inyandiko mubiranga indangamuntu zitunganya mubyukuri, kuringaniza ingano yamagambo nibisobanuro.
Ni ikihe kibazo subword tokenisation ikemura ugereranije no gukoresha amagambo yose?
Amagambo yose yamagambo ni manini kandi aracyabura amagambo adasanzwe; subwords komeza amagambo acungwa kandi neza agabanye amagambo atazwi.
Ninde muribi ni subword tokenisation algorithm ikoreshwa na BERT?
BERT ikoresha WordPiece, ihitamo guhuza byongera imyitozo ya corpus bishoboka.
Nigute WordPiece iranga igice gikomeza ijambo?
WordPiece ibanziriza ijambo-imbere ijambo ryimbere hamwe na '##', bityo 'gukina' bihinduka 'gukina' wongeyeho '## ing'.
Kuki SentencePiece ikwiranye neza nindimi nkikiyapani?
SentencePiece ikora kumyandiko mbisi kandi igashyiraho umwanya nkikimenyetso kidasanzwe, bityo ikora nubwo idafite umwanya hagati yamagambo.
Ni ubuhe buryo buri jambo ryibice byerekana ikarita mbere yo kugera ku cyitegererezo?
Buri subword yahawe indangamuntu yuzuye, iyinjizamo igahinduka vector icyitegererezo gishobora gutunganya.