SentencePiece Tokenization
SentencePiece alama ce ta harshe-agnostic wanda ke koyon yadda ake raba ɗanyen rubutu zuwa guntun kalmomi kai tsaye daga bayanai, ba tare da dogaro da sarari ba.
Dubawa
It made multilingual models far easier to build by treating any language the same way.
Zurfafa nutsewa
Yawancin masu yin alama suna ɗaukar kalmomi suna raba su ta sarari, waɗanda ke raguwa don harsuna kamar Jafananci, Sinanci, ko Thai waɗanda ba sa amfani da su. SentencePiece, wanda Google ya fitar a cikin 2018, ya rabu da wannan ta hanyar ɗaukar shigarwar azaman ɗan rafin haruffa - wuraren da aka haɗa - da koyon ƙamus na raka'o'in kalmomin ƙasa daga bayanan kanta. Ya shahara ya maye gurbin sarari tare da alama mai gani (alamar alama mai kama da meta) don haka alamar ta zama cikakkiyar jujjuyawa: koyaushe kuna iya sake gina ainihin ainihin rubutun. SentencePiece yana goyan bayan manyan algorithms guda biyu, Byte-Pair Encoding (BPE) da ƙirar harshe Unigram, na ƙarshe shine hanyar sa hannu. Saboda ba ya buƙatar takamaiman takamaiman harshe na musamman, bututun guda ɗaya yana aiki a cikin ɗaruruwan harsuna, wanda shine dalilin da ya sa samfura kamar T5, ALBERT, da yawancin tsarin harsuna da yawa suka dogara da shi.
Fahimtar Fasaha
Algorithm na SentencePiece's Unigram yana farawa da ƙaƙƙarfan ƙamus na ɗan takara kuma yana tsattsage gutsuttsura waɗanda ke ba da gudummawa aƙalla ga yuwuwar ƙungiyar horarwa, ta amfani da tsarin Tsammani-Maximization. Alamar sararin samaniya da ake gani (alamar meta) tana ba ta damar yin alama da kuma cirewa ba tare da asara ba. Hakanan yana iya aiki a matakin byte, yana ba da garantin cewa kowane hali - ko da emoji ko rubutun da ba a gani ba - ana iya wakilta ba tare da gazawar ƙamus ba.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar SentencePiece Tokenization
SentencePiece ya kasance dokin aiki don nau'ikan harsuna da yawa saboda jujjuyawar sa da tsaka-tsakin harshe. Filin a hankali yana bincika matakan byte da hanyoyin da ba su da alamar alama waɗanda ke tsallake ƙamus ɗin ƙamus gabaɗaya, da nufin cire ƙamus na tokenization waɗanda ke cutar da lissafi, yarukan da ba kasafai ba, da dogayen lambobi. Ko da haka, SentencePiece's Unigram da ƙirar byte-fallback suna ci gaba da yin tasiri ga sabbin abubuwan tokenizers, kuma rashin asararsa, falsafar rubutu-daga-raw-raw za ta kasance mai tushe nan gaba.
Aiwatar da Gaskiyar Duniya
Google's T5, wanda ke amfani da ƙamus na SentencePiece da aka horar akan rubutun gidan yanar gizo na harsuna da yawa.
Alamar rubutun Jafananci ko na Sinanci waɗanda ba su da sarari tsakanin kalmomi, inda alamun tushen kalma suka gaza.
Gina ƙamus ɗin da aka raba guda ɗaya a cikin yaruka 100+ don tsarin fassarar harsuna da yawa.
Rashin sake gina shigarwar asali (ciki har da tazara) daga alamomi, mai amfani ga ƙirƙira lambar inda abubuwan farin sarari suke.
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the SentencePiece Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Subword Tokenization
Tambayoyin da ake yawan yi
What is SentencePiece Tokenization?
SentencePiece alama ce ta harshe-agnostic wanda ke koyon yadda ake raba ɗanyen rubutu zuwa guntun kalmomi kai tsaye daga bayanai, ba tare da dogaro da sarari ba. Ya sa ƙirar harsuna da yawa sauƙin ginawa ta hanyar kula da kowane harshe iri ɗaya.
Wanne mahimmin zato SentencePiece ke guje wa waccan alamar gargajiya ta dogara da ita?
SentencePiece yana ɗaukar shigarwa azaman rafin haruffa, don haka yana aiki har ma don harsuna ba tare da sarari tsakanin kalmomi ba.
Me yasa SentencePiece ke maye gurbin sarari tare da alamar meta ganuwa?
Rufe sarari azaman alamar bayyane yana nufin rubutun asali, gami da tazara, koyaushe ana iya sake gina su daidai.
Wadanne algorithms guda biyu ne SentencePiece ke tallafawa da farko?
SentencePiece yana ba da Rubutun Byte-Pair (BPE) da samfurin yaren Unigram, tare da Unigram shine hanyar sa hannu.
Ta yaya ƙirar Unigram ke gina ƙamus?
Unigram yana farawa da ɓangarorin ɗan takara da yawa kuma yana kawar da waɗanda ke ba da gudummawa kaɗan ga yuwuwar ƙungiyar ta amfani da Tsari-Maximization.
Wanne samfurin ya shahara yana amfani da alamar SentencePiece?
Google's T5 yana amfani da ƙamus na SentencePiece, kamar yadda ALBERT da samfuran harsuna da yawa suke yi.