Ururimi AI
Ijambo rya WordPiece
WordPiece ni uburyo bwo gutandukanya amagambo mu bice bito hakoreshejwe urutonde rw’amagambo rufite agaciro, bityo amagambo adasanzwe nka "johanson" agahinduka ibice byamenyekanye nka "johan" na "##son".
Umva iki gitaboAI-byavuzwe · ~7 min
Kuriyi page8 min soma
Incamake
Yakozwe kuri Google ku bw'ubushakashatsi bw'ijwi ry'Igiporofita n'Igikoreya mu 2012, byaje kuba tokenizer y'uburyo bwa Google bwo guhindura indimi n'uburyo bwa BERT, kandi byagize uruhare mu buryo bw'imibare y'ubwoko bw'imvugo isoma inyandiko.
Ibyingenzi byingenzi
- WordPiece igabanya amagambo mu bice bito hakoreshejwe urutonde rw’amagambo rufite agaciro, ikerekana ibice byakomeza hakoreshejwe "##" mu buryo bwa BERT.
- Byashyizwe ahagaragara na Schuster na Nakajima muri Google mu 2012 ku isesengura ry’amajwi mu rurimi rw’Igiporofani n’Igikoreya.
- BERT, yatangajwe mu 2018, yakoresheje WordPiece embeddings hamwe n'ijambo ry'ikinyugunyugu riri mu magambo 30,000.
- Mu gihe cyo gusesengura WordPiece ikoresha greedy longest-match-first, kandi ijambo ridafite igice gihuye rihinduka [UNK] yose.
- Nta BPE, WordPiece ifata merges hakurikijwe score ishyira igipimo cy'umubare w'ibice ku mubare w'ibice by'ibice, ntibikurikiza umubare w'ibice gusa.
- Algorithm ya Fast WordPiece ya Google tokenizes mu gihe cya linear kandi yamenyeshejwe ko ari 8.2 inshuro yihuta kurusha Hugging Face Tokenizers.
Kwibira cyane
Tokenization is the first step in any language model: turning raw text into a sequence of units, called tokens, that the model has embeddings for. Splitting on spaces gives whole words, but no fixed word list can cover every name, typo, technical term and inflected form, and every unknown word collapses into a single unknown token. Splitting into characters covers everything but makes sequences long and forces the model to learn spelling from scratch. Subword tokenization sits between the two: common words stay whole, and rare words are broken into smaller pieces that are in the vocabulary. WordPiece is one of the main subword methods, alongside byte-pair encoding (BPE) and the unigram language model used in SentencePiece.
WordPiece comes from Google. Mike Schuster and Kaisuke Nakajima described it in a 2012 ICASSP paper on Japanese and Korean voice search, where text has no spaces between words and a large character set, so a speech recognizer needed a data-driven way to choose units. Google's 2016 neural machine translation paper by Yonghui Wu and colleagues adopted the same model and explained its goal: given a training corpus and a target number of tokens D, pick D wordpieces so the corpus can be segmented into as few pieces as possible, which amounts to maximizing the likelihood of the data under a simple language model. Its example sentence segments Jet makers feud over seat width as _J et _makers _fe ud _over _seat _width, where the underscore marks the start of a word and common words stay whole.
Training a WordPiece vocabulary works much like BPE, by repeatedly merging pieces. Start with every character in the corpus as a token, marking characters that do not begin a word with a prefix; in BERT's convention, word becomes w ##o ##r ##d. Count how often each adjacent pair of tokens appears. Then merge the pair with the highest score and add the result to the vocabulary, repeating until the vocabulary reaches the target size. The difference from BPE is the score. BPE merges the most frequent pair. The WordPiece description in Hugging Face's course divides the pair's frequency by the product of the frequencies of its two parts, which favors pairs whose parts rarely appear apart. The course notes that Google never open-sourced its training code, so this formula is a reconstruction from the literature.
Applying a trained vocabulary to new text is simpler. BERT's tokenizer first normalizes text, converting whitespace and, for uncased models, lowercasing and stripping accents. It then splits off punctuation. Finally it runs WordPiece on each word with greedy longest-match-first: find the longest vocabulary entry that matches the start of the word, emit it, and continue on the rest of the word using pieces with the ## prefix. If at any point no piece matches, the whole word becomes the unknown token [UNK], rather than only the unmatched letters. Only the final vocabulary is stored, not an ordered list of merges, which is another difference from BPE.
BERT made WordPiece famous. The October 2018 paper by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova used WordPiece embeddings with a 30,000-token vocabulary, citing the 2016 translation paper, along with special tokens such as [CLS] at the start of each input, [SEP] between sentences and [MASK] for pretraining. The BERT repository's example shows the whole pipeline: John Johanson's house becomes john johan ##son ' s house. Multilingual BERT extended the idea to 104 languages with a shared vocabulary of 110,000 entries, sampling languages with exponential smoothing so that English was drawn about 100 times more often than Icelandic instead of about 1,000 times.
Compared with alternatives, WordPiece and BPE produce similar vocabularies and similar results; the 2016 translation paper reported that a 32,000-wordpiece model reached 38.95 BLEU on WMT English to French against 37.90 for a word model and 38.01 for a character model, and 24.61 against 23.12 and 22.62 on English to German. The main practical differences are the merge score, the longest-match inference and the whole-word [UNK] fallback. SentencePiece and byte-level BPE, used in many later models, work directly on raw text or bytes and so never need an unknown token, while WordPiece as used in BERT depends on a pre-tokenization step that splits on spaces and punctuation.
WordPiece has known weaknesses. Because it relies on whitespace and punctuation splitting, it needs special handling for languages without spaces; multilingual BERT puts spaces around every Chinese, Japanese Kanji and Korean Hanja character before WordPiece runs. Uncased models strip accents, which the multilingual BERT authors admit can remove meaning in some languages. A word with even one character missing from the vocabulary becomes [UNK] entirely. Subword splits also do not follow morphology: a meaningful unit like un or able may or may not survive as a piece, which the course illustrates by noting that a frequent pair can still score low. And token counts vary by language, so the same sentence can cost many more tokens in a language underrepresented in the vocabulary.
In practice, WordPiece is cheap and well supported. The original BERT checkpoints ship a vocab.txt file mapping each piece to an ID, and changing the vocabulary requires updating the model's vocab_size setting. Libraries such as Hugging Face Tokenizers and TensorFlow Text implement it, and Google's Fast WordPiece algorithm, presented at EMNLP 2021, made tokenization linear in input length. The key operational rule is to use exactly the tokenizer a model was trained with, including its casing setting, since a mismatch silently changes every token ID.
04Urugero rwakazi
Gukora tokenizing y'ijambo mu buryo bw'umuntu hakoreshejwe greedy longest match
Fata urutonde rw'inyuguti: john, johan, ##son, ', s, house, h, ##o, ##u, ##s, ##e na [UNK].
Normalize the sentence John Johanson's house to lowercase and split off punctuation to get john johanson ' s house.
For john, the longest match from the start is john itself, so emit john.
For johanson, the longest match is johan; continue on son with the prefix and match ##son.
Emit ' and s as single pieces, then match house whole.
Now try the word hoax: h matches, then ##o, but no piece starts with ##a, so the entire word becomes [UNK].
Ibyo byerekana
The sentence tokenizes to john johan ##son ' s house, matching the example in the BERT repository. The hoax case shows the whole-word [UNK] fallback, which you can confirm by running a BERT tokenizer library on the same text.
Gushyira mu bikorwa Isi
Preparing input for BERT, which the 2018 paper says uses WordPiece embeddings with a 30,000-token vocabulary, so John Johanson's house becomes john johan ##son ' s house in the uncased model.
Labeling named entities with BERT by classifying only the first sub-token of each word, the approach the BERT paper used on CoNLL-2003, and mapping labels back with a word-to-token index map.
Serving multilingual text with multilingual BERT, whose cased model covers 104 languages with one shared 110,000-entry WordPiece vocabulary and splits Chinese characters before WordPiece runs.
Speeding up tokenization in production with Google's Fast WordPiece algorithm, which a 2021 EMNLP paper reports is on average 8.2 times faster than Hugging Face Tokenizers on general text.
Guhitamo ubunini bw'ijambo ry'ikinyarwanda ry'ikoranabuhanga ry'ubuvugizi, rishingiye ku gitabo cya Google cy'umwaka wa 2016 cy'ubuvugizi bw'ikoranabuhanga, cyagaragaje ko amagambo 8,000 kugeza kuri 32,000 y'ibice by'ijambo bitanga amanota meza ya BLEU kandi bigatanga ibisubizo byihuse.
Kazoza ka WordPiece Tokenisation
Umwirondoro wa WordPiece ubu ukurikira cyane mu buryo bw'ibikoresho by'ibikoresho bya BERT bikoresha mu gushakisha, gutegura no kumenya ibiranga, mu gihe ibindi bimenyetso by'ubuvugizi bishya bikoresha byte-level BPE cyangwa SentencePiece tokenizers zitanga ibimenyetso bitazwi. Akazi ka tekiniki kuri WordPiece ubwacyo kagaragaje umuvuduko, nk'uko bigaragara muri Google ya 2021 Fast WordPiece algorithm ifite tokenization mu gihe cy'igihe cy'ibanze. Ibibazo byabayeho birahurizwa hamwe n'ibindi byose by'uburyo bwo gutandukanya amagambo. Uburyo bwo kugabanya ubusumbane mu ndimi ni kimwe: inyandiko ya multilingual BERT igaragaza uburyo allocation y'ijambo biterwa n'uko indimi zifashishwa, kandi indimi zitarimo cyane ziba zigabanyijwe mu bice byinshi, byoroheje. Uburyo bwo kwihanganira ni indi, kuko inyuguti imwe idasanzwe ishobora guhindura ijambo ryose ry'ikinyarwanda riba [UNK]. Ese imirimo y'ejo hazaza izakomeza gukoresha vocabularies z'ubwoko bw'ibice by'ijambo cyangwa izahindura ku byte- cyangwa character-level inputs ifite grouping yizewe ni ikibazo cy'ubushakashatsi gishobora gukomeza, aho kuba trend ishyizweho.
Kubishyira mubikorwa
Impamvu ari ngombwa
- Umuvuduko n'igipimo. Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
- Kugera no kugera. Yagura uburyo bwindimi nuburyo bwo gutumanaho.
- Ibyemezo bisobanutse. Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Witondere
- Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
- Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
- Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Intambwe ikurikira
- Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
- Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
- Komeza kugenzura abantu kugenzura ibisubizo byinshi.
- Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Inkomoko no gusoma
- Google Research (ICASSP 2012)Japanese and Korean Voice Search
- arXivGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- arXiv (NAACL 2019)BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- GitHub (Google Research)google-research/bert
- GitHub (Google Research)Multilingual BERT
- Hugging Face LLM CourseWordPiece tokenization
- arXiv (EMNLP 2021)Fast WordPiece Tokenization
Komeza Ubushakashatsi
Ibaruwa y'amakuru y'ubuntu
Ibuka amakuru ya buri munsi ya AI
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Take the WordPiece Tokenization quiz
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Ibibazo bikunze kubazwa
WordPiece Tokenisation ni iki?
WordPiece is a subword tokenization method that splits text into pieces from a fixed vocabulary, so rare words like johanson become known parts such as johan and ##son. Developed at Google for Japanese and Korean voice search in 2012, it became the tokenizer behind Google's neural translation system and BERT, and it shaped how a generation of language models reads text.
Niki prefix '##' yerekana mubisohoka muri WordPiece?
WordPiece iranga ijambo rikomeza hamwe na '##', bityo 'gukina' bihinduka 'gukina' + '## ing', kureka decoder ikubaka imipaka yamagambo.
Hafi yamagambo angana gute WordPiece ikoreshwa na BERT yumwimerere?
BERT ikoresha amagambo ya WordPiece yamagambo agera kuri 30.000 yama subword, iringaniza ikwirakwizwa nubunini bwimbonerahamwe.
Nigute amahame ya WordPiece yo guhuza atandukanye na Encoding ya Byte-Pair?
BPE ihuza ibice byinshi byegeranye, mugihe WordPiece ihuza byombi byongerera corpus amahirwe, bikunda kubiri byombi bihuza ibicuruzwa byibice byabo.
What does a BERT-style WordPiece tokenizer return if it cannot segment a word into vocabulary pieces?
WordPiece tokenizers use known vocabulary pieces and can return an unknown-token marker if no valid segmentation is found. Character fallback is not guaranteed for every tokenizer configuration.
Mugihe cyo gufata umwanzuro, ni gute WordPiece ihitamo uburyo bwo gutandukanya ijambo?
WordPiece yerekana umururumba, ihuza amagambo maremare yinjira guhera kumwanya uriho, hanyuma ugasubiramo kubisigaye.
Komeza wige
Kuyobora
Abandi bayobozi batoranijwe kuriyi ngingo