Nbanye Byte-Pair
Byte-Pair Encoding (BPE) bụ mkpakọ sitere n'ike mmụọ nsọ algọridim na-ewulite okwu site na ijikọ ugboro ugboro ugboro abụọ nke akara.
Nchịkọta
It is the tokenizer behind GPT models, balancing tiny vocabularies of characters against huge vocabularies of whole words.
Ime miri emi
BPE na-amalite site n'imeso ederede dịka usoro nke mkpụrụedemede (ma ọ bụ bytes raw). Ọ na-agụkwa ụzọ abụọ akara ọ bụla dị n'akụkụ, na-ejikọta ụzọ abụọ na-emekarị ka ọ bụrụ akara ọhụrụ, ma na-emegharị nke a ugboro puku kwuru puku. A na-edekọ ngwakọta ọ bụla dịka iwu. Usoro mkpụrụedemede a na-ahụkarị dị ka 'th', 'ing', ma ọ bụ okwu niile na-aga n'ihu jiri nwayọọ nwayọọ na-aghọ otu akara, ebe okwu ndị na-adịghị ahụkebe na-ekewa n'ime obere iberibe. Na mbụ usoro mkpakọ data sitere na 1994, Sennrich et al mere ya na NLP. na 2016 maka ntụgharị igwe. GPT-2 na GPT-4 na-eji ọkwa BPE byte, nke na-arụ ọrụ na UTF-8 bytes ka agwa ọ bụla, emoji, ma ọ bụ asụsụ nwere ike ịdebanye aha mgbe ọ bụla na ọdịda enweghị okwu.
Nghọta nka nka
Ọzụzụ BPE na-ewepụta ndepụta ahaziri nke iwu njikọta. Iji megharịa ederede ọhụrụ, algọridim na-ekewa ya ka ọ bụrụ bytes/mkpụrụedemede ma tinye ya na anyaukwu n'otu usoro dị mkpa ruo mgbe enweghị iwu dabara. BPE dị larịị na-ekwe nkwa ndaghachi azụ: ọbụlagodi akara a na-adịghị ahụ anya na-emebi n'ime bytes mejupụtara ya, yabụ okwu nke 256 bytes gbakwunyere njikọ mmụta mmụta na-ekpuchi ihe niile na-enweghị akara UNK.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke ntinye ntinye Byte-Pair
BPE ka bụ tokenizer na-arụ ọrụ, mana nrụgide na-etolite n'ụdị byte- ma ọ bụ ọkwa agwa nke na-amapụ akara ngosi doro anya, na-ezere quirks dị ka nkewa na-adịghị mma na koodu, mgbakọ na mwepụ, ma ọ bụ edemede Bekee na-abụghị. Nchọpụta n'ime ụlọ arụrụ arụ na-enweghị akara na ihe mmụta tokenizers na-achọ idozi echiche BPE. N'agbanyeghị nke ahụ, ọsọ ya na mkpakọ arụmọrụ pụtara ụdị okwu BPE ga-eme ka ọtụtụ LLM mmepụta maka ọdịnihu dị nso.
Mmejuputa n'ezie n'ụwa
GPT-2 na GPT-4 na-eji ọkwa BPE byte ka enwere ike itinye mkpụrụedemede Unicode ma ọ bụ emoji ọ bụla na-enweghị mmejọ.
Sistemụ ntụgharị asụsụ igwe na-eji BPE kewaa mkpụrụokwu ndị na-adịghị ahụkebe ma ọ bụ ngwakọta n'ime mkpụrụokwu obere okwu enwere ike iji ọzọ na-ekekọrịta n'ofe asụsụ.
Ọbá akwụkwọ tokenizers ịmakụ Face na-azụ okwu BPE maka ngalaba omenala dị ka ederede biomedical ma ọ bụ nke iwu.
Ụdị koodu tokenize njirimara na mkpụrụokwu na BPE, na-ejikọta ụkpụrụ ugboro ugboro dị ka 'def' ma ọ bụ '==' n'ime otu token.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Byte-Pair Encoding quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Tokenization na Byte Pair Encoding
Ajụjụ a na-ajụkarị
What is Byte-Pair Encoding?
Byte-Pair Encoding (BPE) bụ mkpakọ sitere n'ike mmụọ nsọ algọridim na-ewulite okwu site na ijikọ ugboro ugboro ugboro abụọ nke akara. Ọ bụ akara ngosi n'azụ ụdị GPT, na-edozi obere okwu nke agwa megide nnukwu okwu nke okwu niile.
Kedu ihe bụ isi ọrụ BPE na-emegharị iji wuo okwu ya?
BPE na-agụta akara abụọ dị n'akụkụ wee jikọta nke na-emekarị ka ọ bụrụ akara ọhụrụ, na-emegharị ọtụtụ puku ugboro.
Kedu ihe BPE na-emeso ederede dị ka mgbe ọ malitere ọzụzụ?
BPE na-amalite site na nkeji kacha nta (edemede ma ọ bụ raw bytes) ma na-etolite akara ngosi site na njikọta.
Kedu ihe kpatara BPE nke ọkwa byte ji ezere mperi na-abụghị nke okwu (UNK)?
N'ihi na okwu ndị isi na-agụnye ihe niile 256 bytes, ọbụna akara ndị a na-adịghị ahụ anya na-adaghachi na nnochite anya byte ha.
Ebee ka BPE algọridim sitere na mbụ tupu NLP anabata ya?
Ewebata BPE dị ka usoro mkpakọ data na 1994 wee megharịa maka ntinye okwu subword na 2016.
Mgbe ị na-egosi ederede ọhụrụ, kedu ka BPE si etinye iwu ya mụtara?
A na-enye iwu iwu njikọta, na tokenization na-etinye ha n'anyaukwu site na mbụ ruo mgbe enweghị iwu ọzọ dabara.