Tokenisasioŋ WordPiece
WordPiece mooy algorithm biy màndargaal baat yu ndaw yi, mooy dooleel BERT ak model yu bari, di xaaj baat yi ay pàcc yuñ mëna jëfandikoowaat suko defee model bi mëna jëfandikoo bépp mbind bu am vokabileer bu takku.
Résumé
It is why a model that has never seen 'unhappiness' can still understand it by reading 'un', '##happy', and '##ness'.
Plongeur bu xóot
WordPiece dafay tabax vocabulaire bu ay baat yu ndaw, du ay baat yu mat wala benn araf. Tambalee ci benn-benn araf, mu boole ci bëgg-bëgg ñaari màndarga yi gëna yokk mën mënu corpus tàggat, baamtu ba muy yegg ci dayo vocabulaire biñ bëgg (BERT dafay jëfandikoo lu tollu ci 30,000 jetons). Ci inference, dafay tokenizes cammo-ci-ndijoor, méngoo ak subword bi gëna gudd ci vocabulaire bi, ba noppi wéy ci dessitu bi. Piyeesu wéyal yi ci biir baat bi dañu leen màrkee '##' prefix, kon 'playing' dafay nekk 'playing' + '##ing'. Loolu mooy saafara jafe-jafe bi nekk ci bitti vocabulaire: baat yu bari wala yu ñu gisul dañuy seey nekk ay fragment yuñ xam, wàcci ba ci benn araf su ko jaree, waaye baat yu bari yi dañuy des nekk ay token yu benn ngir am njariñ.
Gis-gis xarala
WordPiece wuute na ak Kodage Byte-Pair ci anam wi ñuy boolee. BPE dafay boole peer adjacent bi gëna bari; WordPiece dafay boole ñaar ñi gëna mëna am done yuñ mëna tàggat, mu tànnee ñaar ñi seen fréquence boole gëna ëpp produit bi ci fréquence yi ci pàcc yi. Markeer '##' dafay wuutale piyeesu baat yi ak wéy yi, loolu mooy tax tokenizer bi defaraat baat yi ci anam wu leer suñuy dekodewaat ci bind bi.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu tokenisasioŋ WordPiece
Modèlu làkk yu bees yu mag yi dañu gëna bëgg BPE (famiy GPT) wala modelu unigram SentencePiece, ñoom ñooy moytu jëfandikoo làkk wi bu njëkk ba noppi di jëfandikoo bépp dugal Unicode. WordPiece mingi wéy di nekk royuwaay ci encodeur yu bawoo ci BERT te ba leegi ñu ngi koy jëfandikoo bu baax ngir seetlu ak xaaj. Xaarandi wéyal jëfandikoo gi ci defar NLP, ci wetu gëstu ci byte ak xeetu araf yu amul tokenizer yu mëna mujjee wàññi koolute ci vocabulaire subword fixe yépp.
Doxal ci àdduna dëgg
BERT dafay màndargaal laajte seetlu yi ci Google Seetug, di xaaj baat yi ñu xamul ci ay baat yu ndaw suko defee model bi mëna méngoo ak xët yi am njariñ.
BertTokenizer bu Hugging Face dafay jëfandikoo WordPiece ngir soppi bind bu ñor ci ID token yiñ jox BERT ngir jàngat yëg-yëg ak xàmmee entite bu am tur.
BERT lakk yu bari dafay jëfandikoo vocabulaire WordPiece buñ bokk ci 100+ lakk, loolu mooy tax ñu mëna jëfandikoowaat ay fragment ci script yu wuute.
DistilBERT ak xeetu BERT klinik/biomédikal ñooy moom WordPiece, di jëfandikoo kàddu yu bari yu melni 'pneumonoconiosis' ci xaaj leen ay pàcc yuñ xamee.
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the WordPiece Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Tokenisation sous-baatu
Laaj yi ñuy faral di laaj
What is WordPiece Tokenization?
WordPiece mooy algorithm biy màndargaal baat yu ndaw yi, mooy dooleel BERT ak model yu bari, di xaaj baat yi ay pàcc yuñ mëna jëfandikoowaat suko defee model bi mëna jëfandikoo bépp mbind bu am vokabileer bu takku. Moo tax ku musul gis 'tiis' mën nako xam ba leegi su jàngee 'un', '##happy', ak '##ness'.
Lan la prefix '##' bi di wane ci génnug WordPiece?
WordPiece dafay màndargaal wéyal baat yu ndaw yi ak '##', kon 'fo' dafay nekk 'fo' + '##ing', loolu mooy tax dekodeer bi defaraat ay fësal baat.
Lu tollu ci ban yaatuwaayu vocabulaire WordPiece bi BERT bi njëkka jëfandikoo?
BERT dafay jëfandikoo ab vocabulaire WordPiece bu tollu ci 30,000 subword, di ekilibre coverage bi ak dayo tablo biñ samp.
Lan moo wuutale xeetu boole WordPiece ak Kodage Byte-Pair buñ miin?
BPE dafay boole ñaar ñi gëna bari luñu leen di dendal, waaye WordPiece dafay boole ñaar ñi gëna yokk corpus probabilite, gëna bëgg ñaar ñi seen frequency joint ëpp produit seeni pàcc.
naka la WordPiece di def baat buñu musul gis ci diiru tàggat yaram?
Baatu-baatu yiñ gisul dañu leen di xaaj ay baat-baat yu gëna gudd yuñ xam, ñu dellu ci araf yu benn-benn, loolu mooy saafara jafe-jafe bi nekk ci bitti vocabulaire bi.
Ci jamonoy jël doggal, naka la WordPiece di tànnee ni ñuy xaajalee ab baat?
WordPiece dafay tokenizes ak bëgg-bëgg, méngoo ak duggug vocabulaire bi gëna gudd tàmbalee ci barab bi fi nekk, ba noppi baamtu ci dessitu bi.