Alamar alama
Alamar alama shine matakin da ke yanke rubutu zuwa ƙananan guntu da ake kira alamun, raka'a samfurin harshe yana karantawa da tsinkaya.
Dubawa
Yana tsara farashi, iyakokin mahallin, har ma da yadda samfurin ke sarrafa rubutu da kalmomin da ba a saba gani ba.
Zurfafa nutsewa
Kafin samfurin ya ga rubutun ku, alamar alama tana raba shi zuwa alamomi, waɗanda galibi guntun kalmomi ne maimakon gabaɗayan kalmomi ko haruffa guda. Kalmar 'rashin jin daɗi' na iya zama 'un', 'farin ciki', ko 'tokenization' na iya rabuwa zuwa 'alama' da 'ƙira'. Kalmomin gama-gari galibi suna taswira zuwa alama ɗaya, yayin da ƙananan kalmomi, sunaye, ko lamba suka rabu zuwa da yawa. Ana tsara kowace alama zuwa lambar ID wanda samfurin ya canza zuwa vector. Wannan yana da mahimmanci a zahiri saboda ƙirar suna da ƙayyadaddun windows mahallin da aka auna su cikin alamomi, da lissafin APIs a kowace alama, don haka ƙaƙƙarfan ƙa'idar Turanci ta babban yatsa kusan haruffa 4 ko kalmomi 0.75 kowace alama. Tokenization kuma yana bayanin ƙirar ƙirar ƙira: ƙidayar haruffa ko yin ainihin rubutun yana da wahala saboda ƙirar tana ganin guntu, ba haruffa ɗaya ba.
Fahimtar Fasaha
Yawancin LLMs na zamani suna amfani da alamar ƙaramar kalma kamar Byte Pair Encoding (BPE) ko bambance-bambancen matakin-byte. BPE yana farawa daga haruffa kuma akai-akai yana haɗa nau'i-nau'i masu yawa kusa da su don gina ƙayyadaddun ƙamus (sau da yawa alamun 30,000 zuwa 100,000+). Wannan yana daidaita ma'auni biyu: alamar matakin kalma ba zai iya sarrafa kalmomin da ba a gani ba, yayin da matakin hali yana yin jerin tsayi sosai. Kalmomin ƙananan kalmomi suna ƙyale samfurin ya wakilci kowane kirtani, gami da rubutattun kalmomi da sababbin kalmomi, ta hanyar tsara sanannun guntu, yayin da ake kiyaye jeri a takaice.
Dabarun Tasiri
Shawarwari masu haske
Yana taimaka muku keɓance bayyanannen da'awar fasaha daga harshen talla.
Kudin da kasafin kuɗi
Kuna iya yin mafi kyawun tambayoyin aiwatarwa kafin kashe kuɗi ko lokaci.
Ƙungiya da aikin aiki
Ƙungiyoyin da ke da fahimtar juna suna yin mafi kyawun samfura, manufofi, da yanke shawara na koyo.
Makomar Tokenization
Tokenization yanki ne mai aiki da bincike daidai saboda yana iyakance inganci da gaskiya. Harsunan da ke ba da ɓangarorin ɗimbin yawa suna da tsada kuma suna amfani da mahallin cikin sauri, don haka adalcin yaruka da yawa babban abin damuwa ne a magance shi tare da ingantattun kalmomi masu daidaitawa. Har ila yau, masu binciken suna binciken ƙirar alamar-kyauta ko matakin byte (kamar ByT5) kuma sun koyi ƙayyadaddun alama wanda zai iya cire matakin da aka gyara hannun gaba ɗaya. A yanzu, sa ran manyan ƙamus, mafi wayo na tokenizers na harsuna da yawa, da haɓaka wayar da kan masu amfani game da farashin tushen alami da kasafin kuɗi na mahallin.
Aiwatar da Gaskiyar Duniya
Ana biyan farashin API don samfura kamar GPT da Claude akan kowace shigarwa da alamar fitarwa, don haka ƙidayar alamar tana shafar farashi kai tsaye.
Ana auna iyakoki-taga (misali, 128K ko Alamu 200K) a cikin alamu, suna ɗaukar adadin rubutu ko lambar da zaku iya haɗawa.
Masu haɓakawa suna amfani da tokenizers (kamar tiktoken) don kimanta girman gaggawa da datsa abun ciki kafin aika buƙatun.
Tokenization yana bayanin dalilin da yasa ƙirar ke gwagwarmaya don ƙirga haruffa a cikin kalma ko juyar da kirtani, tunda suna ganin guntun kalmomi, ba haruffa ba.
Hatsari & Tsare-tsare
Ƙungiyoyi daban-daban na iya amfani da kalmar iri ɗaya daban, don haka ayyana iyaka da wuri.
Alamomi na iya yin kama da ƙarfi yayin da aikin zahirin duniya bai yi daidai ba.
Yin watsi da ingancin bayanai da tsare-tsaren kimantawa galibi yana haifar da sakamako mara ƙarfi.
Taswirar Hanya
Fara da ma'anar harshe a sarari na sakamakon da kuke buƙata.
Zaɓi ma'aunin nasara ɗaya da yanayin gazawa ɗaya kafin gwaji.
Gudun ƙaramin matukin jirgi tare da bayanan wakilci, ba saitin demo da aka goge ba.
Daftarin aiki inda Tokenization ke taimakawa kuma inda hanyoyin mafi sauƙi suka fi kyau.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
SentencePiece Tokenization
Tambayoyin da ake yawan yi
Menene Tokenization?
Alamar alama shine matakin da ke yanke rubutu zuwa ƙananan guntu da ake kira alamun, raka'a samfurin harshe yana karantawa da tsinkaya. Yana siffanta farashi a hankali, iyakan mahallin, har ma da yadda ƙirar ke sarrafa haruffa da kalmomin da ba kasafai ba.
Menene 'alama' a cikin mahallin samfurin harshe?
Alamu su ne raka'a tsarin tsari, yawanci guntun kalmomi, wani lokaci gabaɗayan kalmomi ko haruffa guda ɗaya.
Wace dabarar alama ce mafi yawan LLMs na zamani ke amfani da su?
Hanyoyin kalmomi irin su BPE suna haɗa nau'i-nau'i na haruffa akai-akai, daidaita raunin matakan-kalmomi masu tsafta da matakan halayen hali.
Me yasa tokenization ke yin ayyuka kamar kirga haruffa a cikin kalma mai wuya ga LLMs?
Saboda an haɗa rubutu zuwa alamomin kalmomi na ƙasa, ƙirar ba ta fahimtar kowane hali kai tsaye, yana sa ayyukan matakin haruffa su zama masu kuskure.
Me yasa tokenization ke da mahimmanci don ƙimar API da iyakokin mahallin?
Lissafin masu ba da lamuni da tagogin mahallin suna da girma a cikin alamomi, don haka ƙidayar alamar tana fitar da farashin duka da nawa za ku iya haɗawa.
Me yasa jumla ɗaya jumla ɗaya zata iya samun ƙarin alamu a wasu harsuna fiye da na Ingilishi?
Kalmomin da aka horar da su akan Ingilishi suna raba wasu harsuna zuwa ƙarin alamu, haɓaka farashi da cin mahallin cikin sauri.