Subword Tokenization
Alamar kalmomin ƙasa tana raba rubutu zuwa raka'a ƙasa da kalmomi amma girma fiye da haruffa, kamar 'alama' da 'ƙira'.
Dubawa
It is the standard way modern language models turn text into the discrete IDs they actually process, balancing vocabulary size against meaning.
Zurfafa nutsewa
Kalmomi sun yi yawa don ƙididdige su ( ƙamus za su yi girma da rasa kalmomi masu wuyar gaske), yayin da haruffa ɗaya ba su da ma'ana kaɗan kuma suna yin jeri sosai. Alamar kalmomin ƙasa shine daidaitawa: yana kiyaye kalmomi akai-akai gabaɗaya amma yana karya ƙananan kalmomi ko hadaddun kalmomi zuwa guntu masu ma'ana. 'Rashin jin daɗi' na iya zama 'un', 'farin ciki', 'rashin hankali'. Manyan Algorithms sun haɗa da Rubutun Byte-Pair (wanda GPT ke amfani da shi), WordPiece (wanda BERT ke amfani da shi), da Unigram/SentencePiece (wanda T5 ke amfani da shi da yawancin nau'ikan harsuna da yawa). Wannan hanyar tana sarrafa kalmomin da ba a gani da kyau, tana raba guda a cikin kalmomi masu alaƙa ('wasa', 'wasa', 'an kunna'), kuma tana goyan bayan kowane harshe. Kowane taswirar guntuwa zuwa ID na lamba, kuma waɗannan ID ɗin sune abin da ƙirar ƙirar ke jujjuyawa zuwa vectors.
Fahimtar Fasaha
Algorithms daban-daban suna zaɓar ƙananan kalmomi daban-daban: BPE yana haɗe nau'i-nau'i akai-akai zuwa sama, WordPiece yana ɗaukar haɗe-haɗe waɗanda galibi suna haɓaka yuwuwar corpus, kuma Unigram yana farawa da ƙamus na ƙamus da ƙasƙanci waɗanda ba su da lahani. WordPiece yana alamar guntun kalmomi-ciki tare da prefix na '##', yayin da SentencePiece ke ɗaukar sarari azaman alama ta musamman don haka yana aiki kai tsaye akan ɗanyen rubutu ba tare da an riga an raba kan farar sararin samaniya ba, mai kyau ga harsuna ba tare da sarari ba.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar Subword Tokenization
Alamar kalmomin shiga za ta kasance da rinjaye saboda yana da sauri kuma mai karamci, amma rauninsa, rarrabuwar kawuna a cikin lissafi, lamba, da rubutun da ba kasafai ba, tare da rashin daidaiton farashi a cikin harsuna, suna haifar da bincike zuwa matakan byte da ƙira marasa kyauta. Yi tsammanin mafi wayo, mai yuwuwa koyo ko masu daidaitawa da kuma mafi kyawun daidaiton yare don haka ba a hukunta rubutun da ba na Ingilishi ba tare da ƙarin alamomin jimla.
Aiwatar da Gaskiyar Duniya
BERT tana amfani da alamar WordPiece, alamar ci gaba kamar '##ing' don sake gina kalmomin asali.
T5 da yawancin nau'ikan harsuna da yawa suna amfani da SentencePiece, wanda ke sarrafa yaruka marasa sarari kamar Jafananci kai tsaye.
Samfuran taɗi sun raba kalmar fasaha da ba kasafai ba zuwa sanannun guntu maimakon kasawa akan kalmar da ba a sani ba.
Tokenizers suna raba ƙananan kalmomi a cikin 'gudu', 'gudu', da 'mai gudu', suna barin ƙirar ta haɓaka ilimin halittar jiki da inganci.
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Subword Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
SentencePiece Tokenization
Tambayoyin da ake yawan yi
What is Subword Tokenization?
Alamar kalmomin ƙasa tana raba rubutu zuwa raka'a ƙasa da kalmomi amma girma fiye da haruffa, kamar 'alama' da 'ƙira'. Ita ce daidaitacciyar hanyar ƙirar harshe na zamani suna juya rubutu zuwa takamaiman ID ɗin da suke aiwatarwa, daidaita girman ƙamus da ma'ana.
Wace matsala ƙaramar kalmar ƙaramar magana ke warwarewa idan aka kwatanta da amfani da duka kalmomi?
Kalmomi gabaɗaya suna da girma kuma har yanzu suna rasa kalmomi masu wuyar gaske; ƙananan kalmomi suna ba da damar sarrafa ƙamus da raba kalmomin da ba a san su ba cikin alheri.
Wanne daga cikin waɗanne ne algorithm ɗin tokenization na ƙananan kalmomi da BERT ke amfani da shi?
BERT tana amfani da WordPiece, wanda ke zaɓar haɗakarwa waɗanda galibi ke haɓaka yuwuwar ƙungiyar horo.
Ta yaya WordPiece yawanci ke yiwa yanki alama da ke ci gaba da kalma?
WordPiece yana ƙaddamar da ƙananan kalmomi na ciki tare da '##', don haka 'wasa' ya zama 'wasa' da '##ing'.
Me yasa SentencePiece ya dace da yaruka kamar Jafananci?
SentencePiece yana aiki akan ɗanyen rubutu kuma yana ɓoye sarari azaman alama ta musamman, don haka yana aiki koda ba tare da sarari tsakanin kalmomi ba.
Menene kowane guntun kalmomi a ƙarshe ke taswira zuwa kafin isa ga samfurin?
Ana sanya kowace kalmar ƙaramar kalmar ID ta lamba, wanda Layer ɗin da ke haɗawa ta juya zuwa vector wanda ƙirar zata iya aiwatarwa.