WordPiece Tokenization
WordPiece shine algorithm ɗin token ƙayyadaddun kalmomi waɗanda ke ba da ikon BERT da ƙira Google da yawa, yana raba kalmomi zuwa gaɓoɓin da za a sake amfani da su ta yadda samfurin zai iya sarrafa kowane rubutu tare da ƙayyadaddun ƙamus.
Dubawa
It is why a model that has never seen 'unhappiness' can still understand it by reading 'un', '##happy', and '##ness'.
Zurfafa nutsewa
WordPiece yana gina ƙamus na raka'o'in ƙananan kalmomi maimakon duka kalmomi ko haruffa guda ɗaya. Farawa daga ɗaiɗaikun haruffa, cikin zari yana haɗa alamomin guda biyu waɗanda galibi suna ƙara yuwuwar ƙungiyar horarwa, tana maimaituwa har sai ta kai ga girman ƙamus (BERT yana amfani da alamun kusan 30,000). A cikin ƙididdiga, yana nuna alamar haɗama daga hagu-zuwa-dama, daidai da mafi tsayin kalma a cikin ƙamus, sannan a ci gaba da saura. Ci gaba da guda a cikin kalma ana yiwa alama da prefix '##', don haka 'wasa' ya zama 'wasa' + '##ing'. Wannan yana magance matsalar rashin ƙamus: ƙananan kalmomi ko ganuwa kawai suna ruɗuwa zuwa gaɓoɓin sananniya, zuwa haruffa ɗaya idan an buƙata, yayin da kalmomin gama gari suna zama a matsayin alamomi guda ɗaya don inganci.
Fahimtar Fasaha
WordPiece ya bambanta da Byte-Pair Encoding a cikin ma'aunin haɗin kai. BPE ya haɗu da mafi yawan lokuta maƙwabta; WordPiece yana haɗa nau'ikan biyu waɗanda ke haɓaka yuwuwar-bayanan horo, kusan zaɓar nau'ikan waɗanda mitar haɗin gwiwa suka fi wuce samfuran mitocin sassanta. Alamar '##' tana bambanta ɓangarorin farko-kalmomi daga ci gaba, yana barin alamar ta sake gina iyakokin kalma ba tare da wata shakka ba yayin yanke hukunci zuwa rubutu.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar WordPiece Tokenization
Sabbin manyan nau'ikan harshe suna ƙara fifita matakin BPE (iyalin GPT) ko ƙirar jumlaPiece unigram, waɗanda ke guje wa takamaiman harshe da sarrafa kowane shigarwar Unicode. WordPiece ya kasance mai tushe a cikin maƙallan da aka samo daga BERT har yanzu ana tura ko'ina don bincike da rarrabuwa. Yi tsammanin ci gaba da amfani a cikin samarwa NLP, tare da bincike cikin byte marasa kyauta da ƙila waɗanda a ƙarshe zasu iya rage dogaro ga ƙayyadaddun ƙayyadaddun ƙamus na ƙayyadaddun kalmomi gaba ɗaya.
Aiwatar da Gaskiyar Duniya
BERT tana ba da alamar tambayoyin bincike a cikin Google Bincika, karya kalmomin da ba a sani ba zuwa cikin ƙananan kalmomi don har yanzu samfurin ya dace da shafukan da suka dace.
Hugging Face's BertTokenizer yana amfani da WordPiece don musanya ɗanyen rubutu zuwa ID ɗin alama da aka ciyar da BERT don nazarin ji da kuma tantance sunan mahaɗan.
BERT na harsuna da yawa yana amfani da ƙamus ɗin WordPiece da aka raba a cikin harsuna sama da 100, yana barin a sake amfani da gutsuttsura a cikin rubutun da ke da alaƙa.
DistilBERT da bambance-bambancen BERT na asibiti/biomedical sun gaji WordPiece, suna sarrafa kalmomin da ba kasafai ake samun su ba kamar 'pneumonoconiosis' ta hanyar raba su cikin sanannun guda.
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the WordPiece Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Subword Tokenization
Tambayoyin da ake yawan yi
What is WordPiece Tokenization?
WordPiece shine algorithm ɗin token ƙayyadaddun kalmomi waɗanda ke ba da ikon BERT da ƙira Google da yawa, yana raba kalmomi zuwa gaɓoɓin da za a sake amfani da su ta yadda samfurin zai iya sarrafa kowane rubutu tare da ƙayyadaddun ƙamus. Shi ya sa samfurin da bai taɓa ganin 'rashin jin daɗi' ba zai iya fahimtarsa ta hanyar karanta 'un', '##happy', da '##ness'.
Menene prefix '##' ke nunawa a cikin fitarwar WordPiece?
WordPiece yana sanya alamar ci gaba da kalmar '##', don haka 'wasa' ya zama 'wasa' + '##ing', yana barin mai yankewa ya sake gina iyakokin kalma.
Kusan girman girman ƙamus na WordPiece da ainihin BERT ke amfani da shi?
BERT tana amfani da ƙamus na WordPiece na kusan raka'o'in ƙananan kalmomi 30,000, daidaita ɗaukar hoto akan girman tebur.
Ta yaya ma'aunin haɗewar WordPiece ya bambanta da daidaitattun Rubutun Byte-Pair?
BPE yana haɗe mafi yawan maƙwabtan maƙwabta, yayin da WordPiece ya haɗu da biyun waɗanda galibi ke haɓaka yuwuwar corpus, suna fifita nau'i-nau'i waɗanda mitar haɗin gwiwa ya wuce samfurin sassansu.
Ta yaya WordPiece ke sarrafa kalmar da ba a taɓa gani ba yayin horo?
Kalmomin da ba a gani sun kasu cikin mafi dadewa da suka dace da ƙananan kalmomin da aka sani, suna komawa zuwa haruffa guda ɗaya, wanda ke magance matsalar ƙamus.
A lokacin tantancewa, ta yaya WordPiece ke zaɓar yadda ake raba kalma?
WordPiece yana nuna alamar zari, yana daidaita mafi tsayin shigarwar ƙamus yana farawa daga matsayi na yanzu, sannan yana maimaita kan saura.