HejuruUbuyobozi bukurikira
Kubogama kuburinganire mubyitegererezo byururimi
Sosiyete
UMUYOBOZI W'umuryango
Itandukaniro rya Tokenisation risobanura ubutumwa bumwe bushobora gufata ibimenyetso byinshi mundimi zimwe kuruta icyongereza, kubera ko ibimenyetso byinshi byubatswe mubyanditswe ahanini byicyongereza.
Umva iki gitaboAI-byavuzwe · ~3 min
Kubera ko serivisi za AI zishyurwa nikimenyetso kandi zikagabanya umubare wibimenyetso bikwiranye nidirishya ryimiterere, abavuga izo ndimi barashobora kwishyura byinshi, bagahuza inyandiko nke bagategereza igihe kirekire kumurimo umwe. Ibyo bituma tokenizer ishushanya ikibazo cyuburinganire kimwe nubuhanga.
A tokenizer yiga ibice byinyandiko bifata nkibimenyetso kimwe wiga corpus. Imirongo isanzwe ibona ibimenyetso byayo; imirongo idasanzwe yacitsemo ibice bito, rimwe na rimwe bikamanuka kuri byite. Iyo iyo corpus yiganjemo icyongereza nizindi ndimi nkeya-zifite indimi nyinshi, izo ndimi zibona ibimenyetso bifatika kumagambo yose hamwe nibice bigize ijambo. Indimi zagaragaye gake cyane, cyangwa zikoresha inyandiko aho buri nyuguti ifata byite nyinshi muri kodegisi ya UTF-8, nka Aziya yepfo yepfo, Amajyepfo yuburasirazuba bwa Aziya na Afrika, byacitsemo ibice byinshi. Ingaruka zirapimwa. Ubushakashatsi bwatangajwe mu 2023, harimo n’ibikorwa byakozwe na Petrov na bagenzi be ku karengane ka tokenizer, bwerekanye ko guhindura inyandiko imwe bishobora gukenera ibimenyetso byinshi mu ndimi zimwe kuruta icyongereza, aho icyuho kinini kirenga inshuro icumi ku ndimi zimwe na zimwe. Ibi bifite akamaro muburyo butatu. Igiciro: Igiciro cya API kuri buri kimenyetso, bityo kubara birebire birebire ni izamuka ryibiciro bitaziguye kubintu bimwe. Context: Idirishya rihamye rifite inyandiko nkeya, bityo inyandiko zigomba kugabanywa kandi ibiganiro bigatakaza impinduka kare vuba. Umuvuduko nubuziranenge: ibimenyetso byinshi bisobanura intambwe yibisekuru byinshi, kandi gucamo ibice biremereye birashobora gutuma bigora icyitegererezo kwiga neza amagambo, bishobora kugira uruhare mukudakora neza murizo ndimi. Abatanga ibisubizo basubije igice. Igihe OpenAI yasohokaga GPT-4o mu 2024, yazanye tokenizer nini kandi ivuga ko yakoresheje ibimenyetso bike mu ndimi nyinshi zitari Icyongereza. Amagambo manini hamwe namakuru aringaniza yimyitozo ifasha, ariko ntibikuraho icyuho rwose. Igitekerezo gikocamye ni uko indimi zimwe zikoreshwa gusa. Ubuhinduzi buratandukanye muburebure, ariko icyuho kinini cyibimenyetso gituruka ahanini kuburyo tokenizer yubatswe, ntabwo biva mundimi ubwazo.
Ubuzima budaharanira inyungu busobanura igitabo kimwe cy’abarwayi mu Cyongereza, Igihinde na Amharic ugasanga verisiyo itari Icyongereza ikoresha ibimenyetso byinshi inshuro nyinshi, bityo fagitire ya API kuri izo ndimi iri hejuru cyane.
Ikiganiro cyabavuga Ikirundi kivuga imipaka nyuma yikiganiro gito ugereranije nicyongereza, bityo ikibagirwa ubutumwa bwambere.
Iterambere rigenzura ingingo yo gushyigikira ikinyakoreya hamwe na tokenizer ireba akabona amagambo menshi agabanyijemo ibice bigufi, mugihe umwimerere wicyongereza ahanini ikarita kumurongo umwe kuri buri jambo.
Itsinda ryubushakashatsi rigereranya ibyashaje nibimenyetso bishya biva mubitanga bimwe ugasanga ibishya bikoresha ibimenyetso bike bigaragara kubimenyetso byabo byicyarabu na Tamil.
Amagambo ya Tokenizer yagiye yiyongera, kandi abatanga serivisi benshi bamenyekanisha kunoza inyandiko zitari Icyongereza, bityo ikinyuranyo gishobora gukomeza kugabanuka ku ndimi zivugwa cyane. Indimi nkeya-indimi zirashobora gukomeza gutinda keretse amahugurwa yamakuru hamwe na tokenizer igishushanyo nkana. Ubushakashatsi kuri byte-urwego kandi bigizwe na moderi bigezweho bishobora kugabanya gushingira kumagambo ahamye, nubwo ibyo bishushanyo bifite ibiciro byabyo. Kugeza ubu, amashyirahamwe akorera abantu benshi bavuga indimi nyinshi agomba gupima umubare wibimenyetso muri buri rurimi bashyigikiye kandi bagafata ikiguzi cyururimi hamwe nimbibi nkurwego rwo gutegura igenamigambi.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Itandukaniro rya Tokenisation risobanura ubutumwa bumwe bushobora gufata ibimenyetso byinshi mundimi zimwe kuruta icyongereza, kubera ko ibimenyetso byinshi byubatswe mubyanditswe ahanini byicyongereza. Kubera ko serivisi za AI zishyurwa nikimenyetso kandi zikagabanya umubare wibimenyetso bikwiranye nidirishya ryimiterere, abavuga izo ndimi barashobora kwishyura byinshi, bagahuza inyandiko nke bagategereza igihe kirekire kumurimo umwe. Ibyo bituma tokenizer ishushanya ikibazo cyuburinganire kimwe nubuhanga.
Tokenizers itanga ibimenyetso bifatika kumirongo isanzwe mumyitozo yabo. Indimi zidahagarariwe zigabanyijemo byinshi, uduce duto.
Buri nyuguti muriyi nyandiko ni UTF-8 bytes nyinshi. Niba tokenizer idakunze guhuza ibyo byte bikurikiranye, inyuguti imwe irashobora guhinduka ibimenyetso byinshi.
Ubuyobozi butondekanya ikiguzi, umwanya uhuriweho, n'umuvuduko cyangwa ubuziranenge. Kwanga byikora ntabwo ari ingaruka zo kubara ibimenyetso.
Uburumbuke bupima ibimenyetso kuri buri jambo. Uburumbuke bukabije bivuze ko inyandiko igabanijwemo ibice byinshi kandi igura byinshi.
GPT-4o yazanye ibimenyetso binini byerekana ibimenyetso OpenAI yavuze ko ikora neza mu ndimi nyinshi zitari Icyongereza.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
Kubogama kuburinganire mubyitegererezo byururimi
Sosiyete