Byte-Byombi Encoding
Byte-Pair Encoding (BPE) ni compression-ihumeka algorithm yubaka amagambo muguhuza inshuro nyinshi ibimenyetso byinshi.
Incamake
It is the tokenizer behind GPT models, balancing tiny vocabularies of characters against huge vocabularies of whole words.
Kwibira cyane
BPE itangira ifata inyandiko nkurukurikirane rwinyuguti zitandukanye (cyangwa byite mbisi). Hanyuma ibara buri kimenyetso cyegeranye, gihuza inshuro nyinshi mukimenyetso gishya, kandi gisubiramo inshuro ibihumbi. Buri guhuza byanditswe nkitegeko. Inyuguti zisanzwe zikurikirana nka 'th', 'ing', cyangwa amagambo yose yakunze guhinduka buhoro buhoro ikimenyetso kimwe, mugihe amagambo adasanzwe akomeza kwigabanyamo uduce duto. Ubusanzwe uburyo bwo gukusanya amakuru kuva 1994, bwahujwe na NLP na Sennrich n'abandi. muri 2016 yo guhindura imashini. GPT-2 na GPT-4 bakoresha byte-urwego BPE, ikorera kuri UTF-8 bytes kuburyo inyuguti iyo ari yo yose, emoji, cyangwa ururimi bishobora guhora byanditseho zeru zitari mu magambo.
Ubushishozi
Amahugurwa BPE itanga urutonde rwateganijwe rwo guhuza amategeko. Kugaragaza inyandiko nshya, algorithm igabanyijemo bytes / inyuguti kandi igashyira hamwe umururumba muburyo bumwe bwambere kugeza igihe nta tegeko rihuye. Urwego rwa Byte urwego BPE rwemeza gusubira inyuma: ndetse nikimenyetso kitagaragara kibora muri byite yacyo, bityo amagambo ya 256 bytes wongeyeho guhuza yize akubiyemo ibintu byose nta kimenyetso cya UNK.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza ka Byte-Byombi Encoding
BPE ikomeza kuba umukoresha wa tokenizer, ariko igitutu kiragenda cyiyongera kuri byte- cyangwa imiterere-urwego rwerekana urugero rwerekana ibimenyetso, birinda gusebanya nko gutandukana nabi muri code, imibare, cyangwa inyandiko zitari Icyongereza. Ubushakashatsi mubyubatswe byubusa kandi byize tokenizers bigamije gukosora kubogama kwa BPE. Nubwo bimeze bityo, umuvuduko wacyo no guhonyora bisobanura BPE-imvugo yamagambo azatanga umusaruro mwinshi LLMs mugihe cya vuba.
Gushyira mu bikorwa Isi
GPT-2 na GPT-4 koresha byte-urwego BPE kugirango inyuguti iyo ari yo yose ya Unicode cyangwa emoji ishobora gushyirwaho kode nta makosa.
Sisitemu yo guhindura imashini ikoresha BPE kugirango igabanye amagambo adasanzwe cyangwa yunganiwe mu magambo yongeye gukoreshwa asanganywe mu ndimi.
Guhobera Isomero rya tokenizers isomero ritoza amagambo ya BPE kuri domaine yihariye nka biomedical or text legal.
Moderi yimyandikire yerekana ibiranga nijambo ryibanze hamwe na BPE, guhuza imiterere kenshi nka 'def' cyangwa '==' mubimenyetso bimwe.
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Byte-Pair Encoding quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Tokenisation na Byte Pair Encoding
Ibibazo bikunze kubazwa
What is Byte-Pair Encoding?
Byte-Pair Encoding (BPE) ni compression-ihumeka algorithm yubaka amagambo muguhuza inshuro nyinshi ibimenyetso byinshi. Nibimenyetso byerekana inyuma ya GPT, kuringaniza amagambo mato yinyuguti arwanya amagambo manini yamagambo yose.
Niki gikorwa cyibanze BPE isubiramo kugirango yubake amagambo?
BPE ibara ibimenyetso bibiri byegeranye kandi igahuza imwe ikunze kugaragara mukimenyetso gishya, gisubiramo inshuro ibihumbi.
BPE ifata iki inyandiko nkigihe itangiye imyitozo?
BPE itangirira ku bice bito (inyuguti cyangwa byite mbisi) kandi ikura ibimenyetso binini binyuze mu guhuza.
Kuki byte-urwego BPE yirinda amakosa-yamagambo (UNK)?
Kuberako amagambo shingiro arimo 256 bytes zose, ndetse nibimenyetso bitagaragara bigaruka kuri byte byerekana.
Algorithm ya BPE yaturutse he mbere yuko NLP ibyemeza?
BPE yatangijwe nkubuhanga bwo gukusanya amakuru muri 1994 nyuma iza guhuzwa na subword tokenisation muri 2016.
Mugihe cyo kwerekana inyandiko nshya, BPE ikoresha ite amategeko yize?
Guhuza amategeko byateganijwe, kandi tokenisation irabikoresha umururumba byihutirwa kugeza ntayandi mategeko ahuye.