InoteveraGaidhi rinotevera
Rusaruro rweGender muMienzaniso yeMitauro
Nzanga
MUTUNGAMIRIRO weSosaiti
Kusiyanisa kweTokenization kunoreva kuti meseji imwe chete inogona kutora akawanda matokens mune mimwe mitauro kupfuura muChirungu, nekuti ma tokenizer mazhinji akavakwa kubva muzvinyorwa zvinonyanya Chirungu.
Teerera kugwaro iriAI-yakarondedzerwa · ~3 min
Sezvo masevhisi eAI achichaja nechiratidzo uye achidzikamisa kuti matani ma tokens anokodzera muhwindo remukati, vatauri vemitauro iyoyo vanogona kubhadhara zvakanyanya, kukwana zvinyorwa zvishoma uye kumirira kwenguva yakareba basa rimwe chete. Izvo zvinoita kuti tokenizer dhizaini ive nyaya yerunako pamwe neyehunyanzvi.
Tokenizer inodzidza kuti ndeapi machunks emavara ekubata senge tokeni nekudzidza kosi yekudzidzira. Tambo dzakajairika dzinowana zviratidzo zvadzo; tambo dzisingawanzo damburwa kuita zvidimbu zvidiki, dzimwe nguva zvichidzika kune ega mabheti. Kana iyo corpus ichidzorwa neChirungu nemimwe mitauro mishoma-yepamusoro-soro, mitauro iyoyo inowana ma tokens emazwi akazara nezvikamu zvezwi zvakajairika. Mitauro isinganyanyi kuoneka, kana inoshandisa zvinyorwa apo mavara ega ega anotora mabhayithi akati wandei muUTF-8 encoding, senge manyoro akawanda ekuSouth Asia, Southeast Asia neAfrica, anopatsanurwa kuita zvimwe zvidimbu zvakawanda. Mhedzisiro yacho inogona kuyerwa. Tsvagiridzo yakaburitswa muna 2023, kusanganisira basa naPetrov nevamwe vaaishanda navo pane tokenizer kusarongeka, yakaona kuti shanduro dzechinyorwa chimwe chete dzinogona kudiwa kakati wandei mune mimwe mitauro kupfuura muChirungu, paine mikaha mikuru inodarika kagumi kune mimwe mitauro uye zviratidzo. Izvi zvinoshanda munzira nhatu dzakasimba. Mutengo: API mitengo ndeye chiratidzo, saka iyo yakareba tokeni kuverenga ndeye yakananga mutengo wekuwedzera kune zvakafanana zvirimo. Mamiriro ezvinhu: hwindo remukati rakagadziriswa rinobata mashoma chaiwo mavara, saka zvinyorwa zvinofanirwa kudimburwa uye nhaurirano dzinorasikirwa nekutanga kutendeuka nekukurumidza. Kumhanya uye kunaka: zviratidzo zvakawanda zvinoreva nhanho dzechizvarwa, uye kupatsanurwa kunorema kunogona kuita kuti zviomere kumuenzaniso kudzidza mamiririro akanaka emazwi, izvo zvinogona kuita kuti kusashanda zvakanaka mumitauro iyoyo. Vanopa rubatsiro vakapindura zvishoma. OpenAI payakaburitsa GPT-4o muna 2024, yakaunza tokenizer yakakura uye yakati yakashandisa tokeni shoma kumitauro yakawanda isiri yeChirungu. Mazwi akakura uye akawedzera kudzidziswa data anobatsira, asi haabvise gaka zvachose. Pfungwa isiriyo yakajairika ndeyokuti mimwe mitauro inongova mashoko. Shanduro dzinosiyana pakureba, asi mikaha mikuru yechiratidzo inouya zvakanyanya kubva mumagadzirirwo echiratidzo, kwete kubva mumitauro pachayo.
Sangano rehutano risingabatsiri rinoshandura bepa remurwere rimwechete muChirungu, Hindi neAmharic uye rinowana zvinyorwa zvisiri zveChirungu zvinoshandisa mamwe ma tokens kakawanda, saka API yayo ye API yemitauro iyoyo yakakwira zvakanyanya.
Chatbot yevatauri veBurmese inosvika pakugumira mushure mekutaura kushoma kwazvo kupfuura shanduro yeChirungu, saka inokanganwa mameseji ekutanga nekukurumidza.
Mugadziri anotarisa chinyorwa chekutsigira cheKorea nechitaridzi chechiratidzo uye anoona mazwi mazhinji akatsemurwa kuita zvidimbu zvipfupi, nepo Chirungu chepakutanga chinoburitsa chiratidzo chimwe pashoko.
Chikwata chekutsvagisa chinofananidza chekare uye chitsva chechiratidzo kubva kumupi mumwe chete uye chinowana mutsva anoshandisa zvinooneka mashoma ma tokeni kune avo echiArabic neTamil.
Mazwi eTokenizer anga achikura, uye vatapi vakati wandei vava kushambadza kuvandudzwa kwemavara asiri echiRungu, saka gaka rinogona kuramba richidzikira pamitauro inotaurwa nevakawanda. Mitauro yakaderera-zvishandiso inogona kuramba ichinonoka kunze kwekunge dhata rekudzidzisa uye tokenizer dhizaini ichisanganisira nemaune. Tsvagiridzo yemhando ye-byte-level uye ine simba mumapoka inogona kuderedza kutsamira pane yakagadziriswa mazwi, kunyangwe iwo madhizaini aine iwo ega compute mutengo. Parizvino, masangano anoshandira vateereri vemitauro yakawanda anofanirwa kuyera nhamba dzematokeni mumutauro wega wega wavanotsigira uye kubata mutengo wemutauro wega wega uye mamiriro ezvinhu sechikamu chekuronga zvekuwanikwa.
Tsamba yemahara
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Kusiyanisa kweTokenization kunoreva kuti meseji imwe chete inogona kutora akawanda matokens mune mimwe mitauro kupfuura muChirungu, nekuti ma tokenizer mazhinji akavakwa kubva muzvinyorwa zvinonyanya Chirungu. Sezvo masevhisi eAI achichaja nechiratidzo uye achidzikamisa kuti matani ma tokens anokodzera muhwindo remukati, vatauri vemitauro iyoyo vanogona kubhadhara zvakanyanya, kukwana zvinyorwa zvishoma uye kumirira kwenguva yakareba basa rimwe chete. Izvo zvinoita kuti tokenizer dhizaini ive nyaya yerunako pamwe neyehunyanzvi.
Tokenizers inopa ma tokeni anoshanda kune tambo dzakajairika mune yavo yekudzidzira corpus. Mitauro isingamiririrwe inopatsanurwa kuita zvimwe, zvidimbu zvidiki.
Hunhu hwega hwega mune zvinyorwa izvi akati wandei UTF-8 bytes. Kana iyo tokenizer isingawanzo kubatanidza iyo byte kutevedzana, hunhu humwe hunogona kuve akawanda tokeni.
Gwaro rinoratidza mutengo, nzvimbo yemamiriro ezvinhu, uye kumhanya kana kunaka. Kuramba otomatiki hakusi mhedzisiro yekuverenga tokeni.
Kubereka zviyero zviratidzo pazwi. Kubereka kwepamusoro kunoreva kuti zvinyorwa zvinopatsanurwa kuita zvidimbu zvakawanda uye zvinodhura zvakanyanya.
GPT-4o yakauya netokenizer yakakura yemazwi iyo OpenAI yakati yainyatso shanda kumitauro yakawanda isiri yeChirungu.
Ramba uchidzidza
Mamwe madhairekitori akasarudzirwa nyaya iyi
InoteveraGaidhi rinotevera
Rusaruro rweGender muMienzaniso yeMitauro
Nzanga