Tokenization subword
Token nke obere okwu na-ekewa ederede ka ọ bụrụ nkeji pere mpe karịa mkpụrụokwu mana buru ibu karịa mkpụrụedemede, dị ka 'token' gbakwunyere 'nhazi'.
Nchịkọta
It is the standard way modern language models turn text into the discrete IDs they actually process, balancing vocabulary size against meaning.
Ime miri emi
Okwu dị ọtụtụ nke a na-apụghị ịgụta ọnụ (okwu ga-abụ nnukwu ma na-atụfu okwu ndị na-adịghị ahụkebe), ebe otu mkpụrụedemede na-enwe ntakịrị ihe ọ pụtara ma na-eme usoro dị ogologo ogologo. Tokenization subword bụ nkwekọrịta: ọ na-edobe okwu ugboro ugboro n'ozuzu mana ọ na-agbaji okwu ndị na-adịghị ahụkebe ma ọ bụ mgbagwoju anya n'ime mpekere bara uru. 'Enweghị obi ụtọ' nwere ike ịghọ 'un', 'obi ụtọ', 'ahụ'. Isi algọridim gụnyere Byte-Pair Encoding (nke GPT ji), WordPiece (nke BERT ji mee ihe), na Unigram/SentencePiece (nke T5 na ọtụtụ ụdị asụsụ ọtụtụ). Ụzọ a na-ejizi okwu ndị a na-adịghị ahụ anya nke ọma, na-ekerịta ibe n'ofe okwu ndị emetụtara ('egwu','egwu','egwu'), ma na-akwado asụsụ ọ bụla. Maapụ iberi nke ọ bụla gaa na ID integer, na NJ ndị a bụ ihe oyi akwa ntinye ihe nlereanya na-agbanwe ka ọ bụrụ vectors.
Nghọta nka nka
Algọridim dị iche iche na-ahọrọ obere okwu dị iche iche: BPE na-ejikọta ụzọ abụọ ugboro ugboro n'okpuru elu, WordPiece na-ahọrọ njikọ nke na-abawanye ohere nke corpus, na Unigram na-amalite site na nnukwu okwu ma na-akwacha akara ndị na-adịghị emerụ ahụ. WordPiece na-eji prefix '##' akara mkpụrụ okwu-ime, ebe SentencePiece na-ewere oghere dị ka akara pụrụ iche nke mere na ọ na-arụ ọrụ ozugbo na ederede ederede n'ebughị ụzọ kewaa na oghere ọcha, dị mma maka asụsụ na-enweghị oghere.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke Tokenization Subword
Tokenization nke obere okwu ga-anọgide na-achị n'ihi na ọ na-adị ngwa ngwa na kọmpat, mana adịghị ike ya, nkewa na-adịghị mma na mgbakọ na mwepụ, koodu, na scripts na-adịghị ahụkebe, gbakwunyere ọnụ ahịa token na-akwụghị ụgwọ n'ofe asụsụ, na-ebute nyocha n'ime ọkwa byte na ụdị enweghị akara. Na-atụ anya ihe nlebara anya nke ọma, enwere ike ịmụta ma ọ bụ na-agbanwe agbanwe yana izi ezi n'asụsụ dị iche iche ka a ghara iji akara ngosi ndị ọzọ na-abụghị nke bekee taa ntaramahụhụ.
Mmejuputa n'ezie n'ụwa
BERT na-eji tokenization WordPiece, na-aka akara n'ihu n'ihu dị ka '##ing' iji wughachi okwu mbụ.
T5 na ọtụtụ ụdị asụsụ dị iche iche na-eji SentencePiece, nke na-ejikwa asụsụ enweghị ohere dịka Japanese ozugbo.
Ụdị nkata na-ekewa okwu teknụzụ a na-adịghị ahụkebe gaa n'iberibe amaara kama ịdaba na okwu amabeghị.
Tokenizers na-ekerịta obere okwu n'ofe 'ọsọ', 'na-agba ọsọ' na 'onye na-agba ọsọ', na-ahapụ ka ihe nlereanya ahụ chịkọta morphology nke ọma.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Subword Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Tokenization nke SentencePiece
Ajụjụ a na-ajụkarị
What is Subword Tokenization?
Token nke obere okwu na-ekewa ederede ka ọ bụrụ nkeji pere mpe karịa mkpụrụokwu mana buru ibu karịa mkpụrụedemede, dị ka 'token' gbakwunyere 'nhazi'. Ọ bụ ọkọlọtọ ka ụdị asụsụ ọgbara ọhụrụ si atụgharị ederede ka ọ bụrụ ID dị iche iche ha na-ahazi, na-edozi oke okwu megide ihe ọ pụtara.
Kedu nsogbu ntọhapụ obere okwu na-edozi ma e jiri ya tụnyere iji okwu niile?
Okwu-okwu dum buru ibu ma ka na-atụfu okwu ndị dị ụkọ; subwords na-eme ka okwu dị mfe ijikwa ma kewaa okwu amaghi ama n'ụzọ amara.
Kedu n'ime ihe ndị a bụ obere okwu tokenization algọridim nke BERT ji?
BERT na-eji WordPiece, nke na-ahọrọ njikọ nke na-abawanye ohere ọzụzụ ụlọ ọrụ.
Kedu ka WordPiece si ejikarị akara na-aga n'ihu otu okwu?
WordPiece na-eji '##' dokwuo obere okwu-ime okwu prefixes, yabụ 'ịkpọ' na-aghọ 'egwu' gbakwunyere '##ing'.
Kedu ihe kpatara SentencePiece ji dabara nke ọma maka asụsụ dịka Japanese?
SentencePiece na-arụ ọrụ na ederede ederede ma tinye oghere dị ka akara pụrụ iche, yabụ ọ na-arụ ọrụ ọbụlagodi na enweghị oghere n'etiti okwu.
Kedu ihe iberibe okwu obere okwu ọ bụla na-emecha maapụ tupu ha eruo ihe nlereanya?
A na-ekenye obere okwu obere okwu ọ bụla ID integer, nke oyi akwa ntinye na-atụgharị ghọọ vector nke ihe nlereanya ahụ nwere ike ịhazi.