Tokenization da Byte Pair Encoding
Tokenization yana raba rubutu zuwa ƙananan raka'a samfurin harshe a zahiri yana karantawa, kuma Byte Pair Encoding (BPE) shine sanannen hanyar gina wannan ƙamus.
Dubawa
It balances having a manageable vocabulary against handling any word the model might encounter.
Zurfafa nutsewa
Samfuran harshe ba sa ganin ɗanyen haruffa ko cikakkun kalmomi - suna ganin alamomi, ID na lamba da aka tsara zuwa guntun rubutu. Zaɓin waɗannan ɓangarorin ciniki ne: ƙamus na matakin kalmomi suna da girma kuma suna shaƙe kalmomin da ba a gani ko kuskure ba, yayin da matakan hali ke yin jeri sosai. Byte Pair Encoding ya bugi tsakiyar ƙasa. An aro daga 1990s data-compressing algorithm, BPE yana farawa daga haruffa ɗaya (ko raw bytes) kuma akai-akai yana haɗa nau'i-nau'i masu yawa a cikin sabon alama, yana haɓaka ƙamus zuwa ƙananan kalmomin gama gari. Kalmomi akai-akai suna zama alamomi guda ɗaya, yayin da ƙananan kalmomi suka rabu zuwa guntuwar da za a sake amfani da su. BPE-matakin Byte, wanda samfuran GPT ke amfani dashi, yana aiki akan ɗanyen bytes don haka zai iya wakiltar kowane rubutu na Unicode - gami da emoji da kowane harshe - ba tare da gazawar ƙamus ba.
Fahimtar Fasaha
Koyarwar BPE mai haɗama ce da mitar-kore. Farawa daga harafin tushe, yana ƙidaya nau'i-nau'i na alamomin da ke kusa da juna a cikin jikin jiki kuma yana haɗa mafi yawan nau'i-nau'i, yana yin rikodin kowane haɗuwa a matsayin doka. Maimaita wannan dubunnan lokuta yana samar da jerin haɗe-haɗe da aka ba da oda da ƙayyadadden ƙamus. A cikin ƙididdiga, ana yin rikodin rubutu ta hanyar amfani da waɗannan ƙa'idodin haɗin kai. Wannan shine dalilin da ya sa alamar ƙidaya ba kasafai ake ƙidayar kalma ba: sarari, babban girma, da kalmomin da ba kasafai ba duk suna canza yadda guntuwar rubutu zuwa alamomi, kuma kalma ɗaya na iya zama alamu da yawa.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Tokenization da Byte Pair Encoding
Tokenization yana ƙarƙashin sake tunani mai aiki. Byte- da nau'ikan matakin hali irin su ByT5, da kuma abubuwan gine-gine masu tasowa marasa kyauta ko 'byte-latent', suna da niyya don sauke ƙayyadaddun ƙamus gaba ɗaya ta yadda ƙirar ke sarrafa kowane shigarwa da kowane harshe iri ɗaya. Har ila yau, masu bincike suna magance daidaiton tokenization - yawancin harsunan da ba na Ingilishi ba da ƙananan kayan aiki a halin yanzu sun fi tsada fiye da alamomin jimla, haɓaka farashi da raguwa mai tasiri. Yi tsammanin abubuwan tokeners da aka kunna don lamba, lissafi, da ma'auni na yaruka da yawa, da ci gaba da gwaje-gwaje don tura iyaka zuwa ga ɗanyen bytes.
Aiwatar da Gaskiyar Duniya
Samfuran GPT da Llama suna amfani da tokenizers irin na BPE don juya faɗakarwa cikin alamun alamun hanyoyin sadarwa.
Ana auna farashin API da iyakoki-taga a cikin alamu, don haka tokenization kai tsaye yana rinjayar farashi da nawa rubutu ya dace.
Karɓar emoji, lamba, da kalmomin da ba kasafai suke da kyau ba ta hanyar raba su zuwa guntun kalmomin da za a iya sake amfani da su ko guntun byte.
Taimakawa harsuna da yawa a cikin ƙira ɗaya ba tare da keɓantaccen ƙamus na kowane harshe ba, ta hanyar rufaffen matakin-byte.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Tokenization and Byte Pair Encoding quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Rubutun Byte-Pair
Tambayoyin da ake yawan yi
What is Tokenization and Byte Pair Encoding?
Tokenization yana raba rubutu zuwa ƙananan raka'a samfurin harshe a zahiri yana karantawa, kuma Byte Pair Encoding (BPE) shine sanannen hanyar gina wannan ƙamus. Yana daidaita samun ƙamus ɗin sarrafawa akan sarrafa kowace kalma da ƙirar zata iya ci karo da ita.
Menene alamar alama ke haifar da samfurin harshe don karantawa?
Samfura suna aiki akan alamu - ID ɗin lamba waɗanda ke tsaye ga haruffa, ƙananan kalmomi, ko kalmomi - maimakon ɗanyen zaren rubutu.
Ta yaya Byte Pair Encoding ke gina ƙamus?
BPE yana farawa daga haruffa ko bytes kuma cikin zari yana haɗa nau'i-nau'i masu yawa da ke kusa, a hankali suna samar da alamun gama gari gama gari.
Wace matsala hanyoyin ƙananan kalmomi kamar BPE ke warwarewa idan aka kwatanta da alamar matakin matakin kalma?
Kalmomin matakin kalmomi sun gaza kan kalmomin da ba a gani; Alamar ƙasidar ƙaƙƙarfan kalmomi tana karya kalmomin da ba kasafai ba zuwa sanannun guntu, guje wa matsalolin ƙamus yayin da ake iya sarrafa ƙamus.
Menene fa'idar matakin-byte BPE, kamar yadda aka yi amfani da su a cikin samfuran GPT?
Yin aiki akan danyen bytes yana nufin kowane shigarwar da za a iya shigar da shi za a iya rufaffen, don haka babu ainihin haruffan da ba a san su ba ba tare da la'akari da harshe ko alama ba.
Me yasa ƙidayar alamar ƙirar ke yawan bambanta da adadin kalmomi a cikin jumla?
Alamar magana na iya karya kalma ɗaya zuwa guntu masu yawa kuma tana kula da tazara da harka, don haka alama ba kasafai ake ƙirga kalma ɗaya ba.