Tokenization nke SentencePiece
SentencePiece bụ tokenizer-agnostic asụsụ nke na-amụta ka esi kewaa ederede n'ime mkpụrụ okwu subword ozugbo site na data, na-adabereghị na oghere.
Nchịkọta
It made multilingual models far easier to build by treating any language the same way.
Ime miri emi
Ọtụtụ tokenizers na-eche na e kewapụrụ okwu site na oghere, nke na-agbaji maka asụsụ dị ka Japanese, Chinese, ma ọ bụ Thai na-adịghị eji ha. SentencePiece, nke Google wepụtara na 2018, kwụpụrụ nke a site n'imeso ndenye ahụ dị ka mmiri iyi nke mkpụrụedemede - oghere gụnyere - yana ịmụta okwu nke nkeji okwu subword site na data n'onwe ya. Ọ na-eji akara ngosi a na-ahụ anya dochie oghere (akara ngosi meta dị ka ihe na-emesi ike) yabụ na-agbanwegharị ya nke ọma: ị nwere ike wughachi ederede izizi ahụ mgbe niile. SentencePiece na-akwado algọridim isi abụọ, Byte-Pair Encoding (BPE) na ụdị asụsụ Unigram, nke ikpeazụ bụ usoro mbinye aka ya. N'ihi na ọ dịghị mkpa ka ọ dị tupu ntule asụsụ akọwapụtara, otu pipeline na-arụ ọrụ n'ofe narị asụsụ, nke mere ụdị dị ka T5, ALBERT, na ọtụtụ usoro asụsụ dị iche iche na-adabere na ya.
Nghọta nka nka
SentencePiece's Unigram algọridim na-amalite site na nnukwu mkpụrụokwu onye ndoro-ndoro anya ma na-akpachapụ anya n'ibe ndị na-enye aka na ohere nke ụlọ ọrụ ọzụzụ, na-eji usoro atụmanya-oke. Ihe nrịbama oghere a na-ahụ anya (akara meta) na-ahapụ ya ka ọ kpachapụta ma mebie ya n'enweghị nsogbu. Ọ nwekwara ike ịrụ ọrụ na ọkwa byte, na-ekwe nkwa na agwa ọ bụla - ọbụlagodi emoji ma ọ bụ edemede a na-adịghị ahụ anya - bụ nke a na-anọchi anya ya na-enweghị ọdịda nke okwu.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke SentencePiece Tokenization
SentencePiece ka bụ nnukwu ọrụ maka ọtụtụ asụsụ yana ụdị koodu n'ihi ngbanwe ya na nnọpụiche asụsụ ya. Ogige ahụ na-eji nke nta nke nta na-enyocha ụzọ-ọkwa byte na enweghị akara tokenizer nke na-awụpụ mkpụrụokwu nke obere okwu kpamkpam, na-achọ iwepu akara ngosi tokenization na-emerụ mgbakọ na mwepụ, asụsụ ndị na-adịghị ahụkebe, na ọnụọgụ ogologo. N'agbanyeghị nke ahụ, SentencePiece's Unigram na atụmatụ byte-fallback na-aga n'ihu na-emetụta ndị ọhụrụ tokenizers, na enweghị mfu ya, ụgbọ oloko-site-raw-ederede ederede ga-anọgide na ntọala maka ọdịnihu dị nso.
Mmejuputa n'ezie n'ụwa
Ụdị T5 Google, nke na-eji okwu okwu SentencePiece zụrụ na ederede webụ ọtụtụ asụsụ.
Ịkpọ akara ederede Japanese ma ọ bụ Chinese nke enweghị oghere n'etiti okwu, ebe akara okwu dabere na-ada.
Iwulite otu okwu ọnụ n'ofe asụsụ 100+ maka usoro ntụgharị asụsụ ọtụtụ asụsụ.
Na-ewughachi ntinye mbụ na-enweghị atụ (gụnyere oghere) site na token, bara uru maka ọgbọ koodu ebe oghere ọcha dị mkpa.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the SentencePiece Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Tokenization subword
Ajụjụ a na-ajụkarị
What is SentencePiece Tokenization?
SentencePiece bụ tokenizer-agnostic asụsụ nke na-amụta ka esi kewaa ederede n'ime mkpụrụ okwu subword ozugbo site na data, na-adabereghị na oghere. O mere ka ụdị asụsụ dị iche iche dịkwuo mfe iwulite site n'imeso asụsụ ọ bụla otu ụzọ ahụ.
Kedu isi echiche nke SentencePiece na-ezere ihe tokenizer ọdịnala na-adabere na ya?
SentencePiece na-ewere ntinye dị ka iyi agwa agwa, yabụ ọ na-arụ ọrụ ọbụlagodi maka asụsụ na-enweghị oghere n'etiti okwu.
Kedu ihe kpatara SentencePiece ji akara meta a na-ahụ anya dochie oghere?
Idobe oghere dị ka akara a na-ahụ anya pụtara ederede izizi, gụnyere oghere, enwere ike ịrụgharị ya ozugbo.
Kedu algọridim abụọ nke SentencePiece na-akwado nke ọma?
SentencePiece na-enye Byte-Pair Encoding (BPE) na ụdị asụsụ Unigram, yana Unigram bụ usoro mbinye aka ya.
Kedu otu ụdị Unigram si ewulite okwu ya?
Unigram na-amalite site na ọtụtụ mpempe akwụkwọ anamachọihe wee wepụrụ ndị na-atụnye obere ihe na corpus site na iji Expectation-Maximization.
Kedu ihe nlereanya na-eji akara SentencePiece tokenizer?
Google's T5 na-eji okwu SentencePiece, dị ka ALBERT na ọtụtụ ụdị asụsụ.