I-WordPiece Tokenization Guide

Uhlolojikelele

It is why a model that has never seen 'unhappiness' can still understand it by reading 'un', '##happy', and '##ness'.

I-Deep Dive

I-WordPiece yakha ulwazimagama lwamayunithi egama elingaphansi kunamagama aphelele noma uhlamvu olulodwa. Kusukela kubalingiswa ngabanye, ngokuhaha ihlanganisa ipheya yezimpawu ezikhulisa kakhulu amathuba okuba ikhophasi yokuqeqeshwa, iphindaphinde ize ifinyelele usayizi wesilulumagama ohlosiwe (i-BERT isebenzisa amathokheni angaba ngu-30,000). Uma kucatshangelwa, ifanekisa ngokuhaha ukusuka kwesokunxele kuye kwesokudla, ifanise igama elingaphansi elide kunawo wonke kusilulumagama, bese iqhubeka kokusele. Izicucu zokuqhubeka phakathi kwegama zimakwe ngesiqalo '##', ngakho 'ukudlala' kuba 'dlala' + '##ing'. Lokhu kuxazulula inkinga yokuphuma kwesilulumagama: amagama ayivelakancane noma angabonakali avele abole abe yizingcezu ezaziwayo, ehle aze afike kuhlamvu olulodwa uma kudingeka, kuyilapho amagama avamile ehlala njengamathokheni awodwa ukuze asebenze kahle.

I-Technical Insight

I-WordPiece ihlukile kokuthi I-Byte-Pair Encoding kumbandela wayo wokuhlanganisa. I-BPE ihlanganisa amapheya aseduze kakhulu; I-WordPiece ihlanganisa ipheya ekhulisa amathuba edatha yokuqeqeshwa, cishe ukukhetha ipheya okuvama kwayo okuhlangene kudlula umkhiqizo wamafrikhwensi wezingxenye zayo. Umaka we-'##' uhlukanisa izingcezu zamagama zokuqala kusukela ekuqhubekeni, uvumela i-tokenizer yakhe kabusha imingcele yegama ngokusobala lapho ihlehlisa ibuyisela umbhalo.

I-Strategic Impact

Isivinini nesikali

Ukugeleza komsebenzi wolimi kungahamba ngokushesha ngaphandle kokudela ukuvumelana.

Finyelela futhi ufinyelele

Yandisa ukufinyelela kuzo zonke izilimi nezitayela zokuxhumana.

Izinqumo ezicacile

Amaqembu angachitha isikhathi esiningi ekwahluleleni kuyilapho i-automation isingatha impinda.

Ikusasa le-WordPiece Tokenization

Amamodeli ezilimi amasha amasha aya ngokuya ethanda i-BPE yeleveli ye-byte (umndeni we-GPT) noma amamodeli e-SentencePiece unigram, agwema ukucubungula kusengaphambili ngolimi oluthile futhi aphathe noma yikuphi okokufaka kwe-Unicode. I-WordPiece isalokhu iyisisekelo kumakhodi asuselwa ku-BERT asasetshenziswa kabanzi ukuze kuseshwe futhi kuhlukaniswe. Lindela ukusetshenziswa okuqhubekayo ekukhiqizeni i-NLP, okuhambisana nocwaningo lwamamodeli angenayo ithokhenizer namamodeli womlingiswa angagcina ehlise ukuthembela kululumagama olungaguquki lwamagama angaphansi ngokuphelele.

Ukuqaliswa Komhlaba Wangempela

I-BERT ibeka amathokheni emibuzweni yosesho kokuthi Google Usesho, ephula amagama angajwayelekile abe amagama amancane ukuze imodeli isakwazi ukufanisa namakhasi ahlobene.

I-Hugging Face's BertTokenizer isebenzisa i-WordPiece ukuguqula umbhalo ongahluziwe ube ama-ID wethokheni anikezwa i-BERT ukuze kuhlaziywe imizwa nokuqashelwa kwebhizinisi.

I-BERT yezilimi eziningi isebenzisa ulwazimagama olwabiwe lwe-WordPiece kuzo zonke izilimi ezingu-100+, ivumela izingcezu ziphinde zisetshenziswe kuyo yonke imibhalo ehlobene.

I-DistilBERT kanye nezinhlobonhlobo ze-BERT zomtholampilo/zezinto eziphilayo zizuza i-WordPiece, iphatha amagama ezokwelapha angajwayelekile njenge-'pneumonoconiosis' ngokuwahlukanisa abe izingcezu ezaziwayo.

Izingozi & Guardrails

Amaqiniso akhonjiwe angafaka ngokuthula imibiko, ukugeleza kosekelo, noma imiphumela yocwaningo.

Ukuzwela okusheshayo kungadala imiphumela engahambisani kuzo zonke izicelo ezifanayo.

Idatha yombhalo ebucayi ingase idalulwe uma izilawuli zokufinyelela zibuthakathaka.

Ukuqalisa Umhlahlandlela

Chaza ifomethi yokuphumayo, ithoni, namazinga wekhwalithi ngaphambi kokukhishwa.

Izimpendulo eziyisisekelo ngemithombo ethembekile noma nini lapho ukunemba kubalulekile.

Gcina indawo yokuhlola isibuyekezo somuntu ukuze uthole imiphumela ephezulu.

Landela amaphethini okuhluleka futhi uqeqeshe kabusha imiyalo noma ukuhamba komsebenzi njalo.

Qhubeka Uhlole

ChatGPT ne-LLMs

Bona ukuthi amamodeli olimi esimanje akhiqiza kanjani futhi acabange.

Funda Umhlahlandlela

Izisekelo ze-NLP

Funda okuyisisekelo kokucubungula ulimi ngemuva kwalawa mathuluzi.

Funda Umhlahlandlela

Free newsletter

Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.

One email each weekday. Unsubscribe in one click. We never sell or share your address.

Test yourself

Take the WordPiece Tokenization quiz

Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.

Qala imibuzo

Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation

Umhlahlandlela olandelayo

I-Subword Tokenization

Funda umhlahlandlela olandelayo Phequlula yonke imibuzo

Imibuzo evame ukubuzwa

What is WordPiece Tokenization?

I-WordPiece iyi-algorithm yokwenza amathokheni yamagama angaphansi enika amandla i-BERT kanye namamodeli amaningi Google, ihlukanisa amagama abe izingcezu ezisebenzisekayo kabusha ukuze imodeli ikwazi ukuphatha noma yimuphi umbhalo ngesilulumagama esingashintshi. Kungakho imodeli engakaze ikubone 'ukungajabuli' isengakuqonda ngokufunda 'un', '##jappy', kanye '##ness'.

Isiqalo esithi '##' sibonisani ekuphumeni kwe-WordPiece?

I-WordPiece imaka ukuqhubeka kwegama elingaphansi ngo-'##', ngakho-ke 'ukudlala' kuba 'dlala' + '##ing', okuvumela isiqophi serekhodi sakhe kabusha imingcele yamagama.

Cishe sikhulu kangakanani isilulumagama se-WordPiece esisetshenziswa i-BERT yasekuqaleni?

I-BERT isebenzisa isilulumagama se-WordPiece esingaba amayunithi wamagama angaphansi angu-30,000, ibhalansisa ukumbozwa ngokumelene nosayizi wethebula lokushumeka.

Ingabe umbandela wokuhlanganisa we-WordPiece uhluke kanjani ekubhaleni ngekhodi okujwayelekile kwe-Byte-Pair?

I-BPE ihlanganisa amapheya asondelene kakhulu, kuyilapho i-WordPiece ihlanganisa ipheya ekhulisa kakhulu amathuba ekhorasi, ivuna amapheya amafrikhwensi awo ahlangene adlula umkhiqizo wezingxenye zawo.

I-WordPiece iliphatha kanjani igama elingakaze libonwe ngesikhathi sokuqeqeshwa?

Amagama angabonakali ahlukaniswa abe amagama amancane afaniswayo amade kakhulu, abuyela emuva kuhlamvu olulodwa, okuxazulula inkinga yokuphuma kwesilulumagama.

Ngesikhathi sokubikezela, i-WordPiece ikhetha kanjani ukuhlukanisa igama?

I-WordPiece yenza amathokheni ngokuhaha, ifanise ukufakwa kwamagama okude kunawo wonke kuqala endaweni yamanje, bese iphinda kokusele.