Ubuyobozi bwa tekiniki

Tokenisation na Byte Pair Encoding

Tokenisation igabanya inyandiko mubice bito urugero rwururimi rusoma, kandi Byte Pair Encoding (BPE) nuburyo buzwi bwo kubaka ayo magambo.

Umva iki gitaboAI-byavuzwe · ~2 min

Kuriyi page2 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Kazoza ka Tokenisation na Byte Pair Encoding
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

It balances having a manageable vocabulary against handling any word the model might encounter.

Kwibira cyane

Ururimi rwindimi ntirubona inyuguti mbisi cyangwa amagambo yose - babona ibimenyetso, indangamuntu yuzuye yashizwe mubice byinyandiko. Guhitamo ibyo bice ni ubucuruzi: ijambo-urwego rwamagambo ni manini kandi aruniga kumagambo atagaragara cyangwa yanditse nabi, mugihe urwego-rwerekana urwego rukurikirana. Byte Pair Encoding ikubita hagati. Yatijwe kuva 1990-data-compression algorithm, BPE itangirira kumuntu kugiti cye (cyangwa byite mbisi) kandi igahuza inshuro nyinshi guhuza ibice byinshi mubimenyetso bishya, bikura amagambo yerekeza kumagambo asanzwe. Amagambo akunze kuba ikimenyetso kimwe, mugihe amagambo adasanzwe yacitsemo ibice bikoreshwa. Byte-urwego BPE, ikoreshwa na moderi ya GPT, ikora kuri bytes mbisi kuburyo ishobora kwerekana inyandiko iyo ari yo yose ya Unicode - harimo emoji n'ururimi urwo arirwo rwose - nta kunanirwa-amagambo.

Gushyira mu bikorwa Isi

Moderi ya GPT na Llama ikoresha BPE-yuburyo bwa tokenizers kugirango ihindure ibisobanuro mubiranga ibimenyetso byerekana urusobe.

Ibiciro bya API hamwe nimbonerahamwe-idirishya ntarengwa bipimirwa mubimenyetso, bityo tokenisation igira ingaruka itaziguye nigiciro hamwe ninyandiko ihuye.

Gukoresha emoji, kode, namagambo adasanzwe ubigabanyijemo ibice byongeye gukoreshwa cyangwa ibice byte.

Gushyigikira indimi nyinshi murugero rumwe rudafite inkoranyamagambo itandukanye kururimi, ukoresheje byte-urwego rwa kodegisi.

Kazoza ka Tokenisation na Byte Pair Encoding

Tokenisation iri gutekereza cyane. Byte- na inyuguti-urwego rwicyitegererezo nka ByT5, hamwe no kugaragara kwubusa cyangwa 'byte-latent' imyubakire, igamije guta amagambo ahamye rwose kuburyo moderi ikora ibyinjijwe nururimi urwo arirwo rwose. Abashakashatsi kandi barimo guhangana n’uburinganire bwerekana ibimenyetso - indimi nyinshi zitari Icyongereza n’umutungo muke kuri ubu zigura ibimenyetso byinshi cyane kuri buri nteruro, kuzamura ibiciro no kugabanya imiterere ifatika. Tegereza ibimenyetso byerekana kode, imibare, nuburinganire bwindimi nyinshi, wongeyeho ubushakashatsi bwakomeje kugirango usubize imbibi ugana kuri byite mbisi.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Tokenization and Byte Pair Encoding quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is Tokenization and Byte Pair Encoding?

Tokenisation igabanya inyandiko mubice bito urugero rwururimi rusoma, kandi Byte Pair Encoding (BPE) nuburyo buzwi bwo kubaka ayo magambo. Iringaniza kugira amagambo acungwa kurwanya ijambo iryo ari ryo ryose icyitegererezo gishobora guhura nacyo.

Niki tokenisation itanga urugero rwururimi rwo gusoma?

Icyitegererezo gikora ku kimenyetso - indangamuntu yuzuye ihagaze ku nyuguti, ijambo, cyangwa amagambo - kuruta imirongo mbisi.

Nigute Byte Pair Encoding yubaka amagambo yayo?

BPE itangirira ku nyuguti cyangwa bytes kandi umururumba uhuza inshuro ebyiri zegeranye, buhoro buhoro ugakora ijambo ryibisobanuro rusange.

Ni ikihe kibazo uburyo bwa subword uburyo nka BPE bukemura ugereranije nijambo-urwego tokenisation?

Ijambo-urwego rwamagambo ananirwa kumagambo atagaragara; subword tokenisation isenya amagambo adasanzwe mubice bizwi, wirinda ibibazo bitari amagambo mugihe ugumya amagambo gucungwa.

Ni izihe nyungu za byte-urwego BPE, nkuko bikoreshwa muri moderi ya GPT?

Gukorera kuri bytes mbisi bivuze ko ibishoboka byose byinjizwa bishobora gushyirwaho kodegisi, kubwibyo rero nta nyuguti zitazwi rwose utitaye ku rurimi cyangwa ikimenyetso.

Ni ukubera iki icyitegererezo cyerekana ibimenyetso akenshi gitandukanya numubare wamagambo mu nteruro?

Subword tokenisation irashobora kumena ijambo rimwe mubice byinshi kandi byunvikana kumwanya hamwe nurubanza, bityo ibimenyetso bibarwa gake gake kubara ijambo.