Tokenization WordPiece
WordPiece bụ obere okwu tokenization algọridim nke na-akwado BERT na ọtụtụ ụdị Google, na-ekewa mkpụrụokwu n'ime mpekere ọzọ enwere ike iji ya mee ihe ka ihe atụ wee nwee ike ijikwa okwu edobere ederede ọ bụla.
Nchịkọta
It is why a model that has never seen 'unhappiness' can still understand it by reading 'un', '##happy', and '##ness'.
Ime miri emi
WordPiece na-ewulite mkpụrụokwu nke nkeji obere okwu karịa mkpụrụokwu dum ma ọ bụ otu mkpụrụedemede. Malite na mkpụrụedemede ọ bụla, ọ na-eji anyaukwu na-ejikọta akara abụọ nke na-abawanye ohere nke ụlọ ọrụ ọzụzụ, na-emegharị ya ruo mgbe ọ ruru nha nke okwu okwu (BERT na-eji ihe dị ka 30,000 tokens). Na ntinye, ọ na-egosi anyaukwu n'aka ekpe gaa n'aka nri, dabara na mkpụrụokwu kacha ogologo n'ime okwu, wee gaa n'ihu na nke fọdụrụ. Edebere iberibe ihe na-aga n'ihu n'ime okwu na prefix '##', yabụ 'ịkpọ' na-aghọ 'egwu' + '##ing'. Nke a na-edozi nsogbu nke na-abụghị nke okwu: okwu ndị na-adịghị ahụkebe ma ọ bụ ndị a na-ahụghị anya na-aghọ naanị n'ime iberibe amaara, ruo otu mkpụrụedemede ma ọ bụrụ na ọ dị mkpa, ebe okwu nkịtị na-adị ka otu akara maka ịrụ ọrụ nke ọma.
Nghọta nka nka
WordPiece dị iche na Byte-Pair Encoding na njikota ya. BPE na-ejikọta ụzọ abụọ dị n'akụkụ; WordPiece na-ejikọta ụzọ abụọ ahụ na-ebuli ohere data-ọzụzụ, na-ahọrọ ụzọ abụọ nke oge nkwonkwo ha kacha karịa ngwaahịa nke akụkụ ya ugboro ugboro. Ihe nrịbama '##' na-ekewa mkpụrụ okwu-mbụ na ọga n'ihu, na-ahapụ ka tokenizer rụgharịa oke okwu n'enweghị mgbagha mgbe ọ na-atụgharị azụ na ederede.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke Tokenization WordPiece
Ụdị asụsụ buru ibu ọhụrụ na-enwewanye mmasị na ọkwa BPE (GPT) ma ọ bụ ụdị unigram nke SentencePiece, nke na-ezere nhazi asụsụ kpọmkwem ma na-ejikwa ntinye Unicode ọ bụla. WordPiece ka bụ ntọala n'ime koodu ntinye BERT ewepụtara ka na-ebuga ebe niile maka ọchụchọ na nhazi. Na-atụ anya ka a na-aga n'ihu na mmepụta NLP, n'akụkụ nyocha n'ime byte na ụdị agwa na-enweghị tokenizer nke nwere ike mechaa belata ịdabere na mkpụrụokwu subword a kapịrị ọnụ kpam kpam.
Mmejuputa n'ezie n'ụwa
BERT na-eme ka ajụjụ ọchụchọ dị na Google Chọọ, na-agbaji okwu ndị a na-amaghị n'ime obere okwu ka ihe nlereanya ahụ ka nwere ike dabara na ibe ndị dị mkpa.
Ịmak Face's BertTokenizer na-eji WordPiece iji tọghata ederede raw ka ọ bụrụ akara ngosi akara e nyere BERT maka nyocha mmetụta na njirimara aha.
BERT multilingual na-eji okwu WordPiece ekekọrịtara n'ofe asụsụ 100+, na-ahapụ ka ejigharịa iberi n'ofe edemede ndị metụtara ya.
DistilBERT na Clinical/Biomedical BERT dị iche iche na-eketa WordPiece, na-ejikwa usoro ahụike na-adịghị ahụkebe dị ka 'pneumonoconiosis' site na kewaa ha n'ime iberibe ama ama.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the WordPiece Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Tokenization subword
Ajụjụ a na-ajụkarị
What is WordPiece Tokenization?
WordPiece bụ obere okwu tokenization algọridim nke na-akwado BERT na ọtụtụ ụdị Google, na-ekewa mkpụrụokwu n'ime mpekere ọzọ enwere ike iji ya mee ihe ka ihe atụ wee nwee ike ijikwa okwu edobere ederede ọ bụla. Ọ bụ ya mere ihe nlereanya nke na-ahụtụbeghị 'enweghị obi ụtọ' ka nwere ike ịghọta ya site n'ịgụ 'un', '##happy' na '##ness'.
Kedu ihe prefix '##' na-egosi na mmepụta WordPiece?
WordPiece na-eji '##' akara n'ihu nke obere okwu, yabụ 'ịkpọ' na-aghọ 'egwu' + '##ing', na-ahapụ onye nrụpụta ka ọ rụgharịa oke okwu.
Kedu ka okwu WordPiece siri buru ibu nke BERT mbụ ji?
BERT na-eji okwu WordPiece nke ihe dị ka nkeji okwu subword 30,000, na-edozi mkpuchi megide nha nke tebụl.
Kedu ka njiko WordPiece si dị iche na ngbanwe Byte-Pair ọkọlọtọ?
BPE na-ejikọta ụzọ abụọ dị n'akụkụ ugboro ugboro, ebe WordPiece na-ejikọta ụzọ abụọ nke na-abawanye ohere nke corpus, na-akwado ụzọ abụọ nke oge nkwonkwo ha karịrị ngwaahịa nke akụkụ ha.
Kedu ka WordPiece si ejikwa okwu a na-ahụtụbeghị n'oge ọzụzụ?
A na-agbaji okwu ndị a na-adịghị ahụ anya n'ime mkpụrụokwu ndị kacha dabara adaba nke ama ama, na-adaghachi na otu mkpụrụedemede, nke na-edozi nsogbu na-enweghị okwu.
N'oge ntinye, kedu ka WordPiece si ahọrọ ka esi kewaa okwu?
WordPiece tokenizes anyaukwu, na-adaba na ntinye okwu kachasị ogologo na-amalite n'ọnọdụ dị ugbu a, wee na-emegharị na nke fọdụrụ.