Rubutun Byte-Pair
Byte-Pair Encoding (BPE) wani algorithm ne na matsawa wanda ke gina ƙamus ta hanyar haɗa alamomin guda biyu akai-akai.
Dubawa
Ita ce alama a bayan samfuran GPT, daidaita ƙananan ƙamus na haruffa da manyan ƙamus na gabaɗayan kalmomi.
Zurfafa nutsewa
BPE yana farawa ta hanyar ɗaukar rubutu azaman jeri na haruffa ɗaya (ko ɗanyen bytes). Sannan yana ƙirga kowane nau'i na alama da ke kusa, yana haɗa mafi yawan lokuta biyu zuwa sabon alama, kuma yana maimaita wannan sau dubbai. Ana yin rikodin kowace haɗuwa azaman ƙa'ida. Jerin haruffa gama-gari kamar 'th', 'ing', ko gabaɗayan kalmomi akai-akai a hankali sannu a hankali suna zama alamomi ɗaya, yayin da ƙananan kalmomi ke rarrabuwa zuwa ƙananan guntu. Asalin hanyar matsawa bayanai daga 1994, an daidaita shi zuwa NLP ta Sennrich et al. a cikin 2016 don fassarar inji. GPT-2 da GPT-4 suna amfani da matakin-byte BPE, wanda ke aiki akan UTF-8 bytes don haka kowane hali, emoji, ko harshe koyaushe ana iya sanya shi tare da rashin gazawar ƙamus.
Fahimtar Fasaha
Horon BPE yana samar da jerin umarni na ƙa'idodin haɗuwa. Don nuna alamar sabon rubutu, algorithm ɗin yana raba shi zuwa bytes/haruffa kuma ana amfani da shi tare da haɗama cikin tsarin fifiko ɗaya har sai babu ƙa'ida da ta dace. BPE-matakin Byte yana ba da garantin koma baya: ko da alamar da ba a gani ba ta lalace zuwa cikin abubuwan da ke tattare da shi, don haka ƙamus na bytes 256 da haɗin gwiwar koya ya ƙunshi komai ba tare da alamar UNK ba.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar Byte-Pair Encoding
BPE ya kasance alamar alamar dokin aiki, amma matsin lamba yana girma zuwa ga byte- ko ƙirar matakin hali waɗanda ke tsallake ƙayyadaddun alama, guje wa ɓangarorin da ba su da kyau a cikin lambar, lissafi, ko rubutun Ingilishi ba. Bincike cikin gine-ginen da ba su da alamar alama da kuma koyan tokenizers na nufin gyara son zuciya na BPE. Duk da haka, saurinsa da ƙarfinsa na matsawa yana nufin ƙamus na salon BPE zai ƙarfafa yawancin samar da LLMs don nan gaba.
Aiwatar da Gaskiyar Duniya
GPT-2 da GPT-4 suna amfani da matakin-byte BPE don haka kowane hali na Unicode ko emoji za a iya ɓoye shi ba tare da kurakurai ba.
Tsarukan fassarar inji suna amfani da BPE don raba kalmomin da ba kasafai ba ko hadaddun kalmomi zuwa guntun kalmomi na sake amfani da su da aka raba a cikin harsuna.
Rungumar Face's tokenizers ɗakin karatu yana horar da ƙamus na BPE don yankunan al'ada kamar rubutun ilimin halitta ko na doka.
Samfuran lambobi suna nuna masu ganowa da kalmomi masu mahimmanci tare da BPE, suna haɗa alamu akai-akai kamar 'def' ko '==' cikin alamun guda ɗaya.
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Byte-Pair Encoding quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Tokenization da Byte Pair Encoding
Tambayoyin da ake yawan yi
Menene Rubutun Byte-Pair?
Byte-Pair Encoding (BPE) wani algorithm ne na matsawa wanda ke gina ƙamus ta hanyar haɗa alamomin guda biyu akai-akai. Ita ce alama a bayan samfuran GPT, daidaita ƙananan ƙamus na haruffa da manyan ƙamus na gabaɗayan kalmomi.
Menene ainihin aikin BPE yana maimaitawa don gina ƙamus?
BPE yana ƙidaya nau'i-nau'i na alamar maƙwabta kuma yana haɗa guda ɗaya mafi yawan lokuta zuwa sabuwar alama, tana maimaita sau dubbai.
Menene BPE ke bi da rubutu kamar lokacin da ya fara horo?
BPE yana farawa daga mafi ƙanƙanta raka'a (halaye ko raw bytes) kuma yana girma manyan alamu ta hanyar haɗuwa.
Me yasa matakin BPE na byte ke guje wa kurakuran da ba a cikin ƙamus (UNK)?
Domin tushen ƙamus ya ƙunshi duk 256 bytes, har ma da alamun da ba a gani ba suna komawa zuwa wakilcin byte.
A ina ne BPE algorithm ya samo asali kafin NLP ta karbe shi?
An gabatar da BPE azaman dabarar matsi da bayanai a cikin 1994 kuma daga baya an daidaita shi don alamar magana a cikin 2016.
Lokacin yin alamar sabon rubutu, ta yaya BPE ke amfani da ƙa'idodin da aka koya?
An ba da umarnin ƙa'idodin haɗakarwa, kuma tokenization yana amfani da su da zari ta fifiko har sai babu wani ƙarin ƙa'ida.