InoteveraGaidhi rinotevera
Subword Tokenization
Mutauro AI
Mutauro AI GUIDE
SentencePiece imutauro-agnostic tokenizer inodzidza kupatsanura mavara akaomeswa kuita zvidimbu zvidiki zvakananga kubva kudata, pasina kuvimba nenzvimbo.
Teerera kugwaro iriAI-yakarondedzerwa · ~2 min
Yakaita kuti mhando dzemitauro yakawanda dzive nyore kugadzira nekubata chero mutauro nenzira imwechete.
Mazhinji ma tokenizer anofunga mazwi akapatsanurwa nenzvimbo, izvo zvinotsemuka kumitauro yakaita seJapan, Chinese, kana Thai isingaashandise. SentencePiece, yakaburitswa na Google muna 2018, inosiya izvi nekubata iyo inopinza seyakaomeswa mavara - nzvimbo dzinosanganisirwa - uye kudzidza duramazwi remadiki mayuniti kubva kudata pacharo. Iyo ine mukurumbira kutsiva nzvimbo neinooneka chiratidzo (iyo underscore-senge meta chiratidzo) saka tokenization inodzoreredzwa zvizere: unogona kugara uchigadzira zvekare iwo chaiwo mavara ekutanga. SentencePiece inotsigira maviri makuru algorithms, Byte-Pair Encoding (BPE) uye Unigram mutauro modhi, iyo yekupedzisira iri nzira yayo yekusaina. Nekuti haidi pre-tokenization yemutauro wakanangana nemutauro, pombi imwe cheteyo inoshanda mumazana emitauro, ndosaka mamodheru akaita seT5, ALBERT, uye masisitimu mazhinji emitauro yakawanda achivimba nawo.
Google's T5 modhi, inoshandisa mazwi eSentencePiece akadzidziswa pamitauro yakawanda yepawebhu.
Kuisa chiratidzo cheJapan kana chiChinese chinyorwa chisina nzvimbo pakati pemashoko, apo izwi-based tokenizers rinokundikana.
Kuvaka izwi rimwe chete rinogovaniswa mumitauro 100+ yeshanduro yemitauro yakawanda.
Kugadzira pasina kurasikirwa kwekutanga kupinza (kusanganisira nzvimbo) kubva kune tokens, inobatsira pakugadzira kodhi uko kune whitespace.
SentencePiece inoramba iri bhiza rekushanda kune emitauro yakawanda uye macode modhi nekuda kwekudzoreredza kwayo uye kusarerekera kwemutauro. Munda uri kuongorora zvishoma nezvishoma nzira dzebyte-level uye tokenizer-yemahara dzinodarika mazwi emazwi zvachose, achivavarira kubvisa tokenization quirks inokuvadza arithmetic, mitauro isingawanzo, uye nhamba refu. Zvakadaro, SentencePiece's Unigram uye byte-fallback dhizaini inoramba ichipesvedzera nyowani tokenizers, uye yayo isingarasikike, chitima-kubva-raw-mavara uzivi hucharamba huri hwaro munguva pfupi iri kutevera.
Tsamba yemahara
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
SentencePiece imutauro-agnostic tokenizer inodzidza kupatsanura mavara akaomeswa kuita zvidimbu zvidiki zvakananga kubva kudata, pasina kuvimba nenzvimbo. Yakaita kuti mhando dzemitauro yakawanda dzive nyore kugadzira nekubata chero mutauro nenzira imwechete.
SentencePiece inobata mapindiro serukova rwemavara, saka inoshanda kunyangwe kumitauro isina nzvimbo pakati pemazwi.
Encoding nzvimbo sechiratidzo chinooneka zvinoreva kuti mavara ekutanga, kusanganisira spacing, anogona kugara achivakwa patsva chaizvo.
SentencePiece inopa Byte-Pair Encoding (BPE) uye Unigram mutauro modhi, uye Unigram iri nzira yayo yekusaina.
Unigram inotanga nezvimedu zvakawanda zvevamiriri uye inobvisa zvakare izvo zvinopa zvishoma kune corpus mukana uchishandisa Tarisiro-Kuwedzera.
Google's T5 inoshandisa mazwi eSentencePiece, sezvinoita ALBERT nemhando dzakawanda dzemitauro yakawanda.
Ramba uchidzidza
Mamwe madhairekitori akasarudzirwa nyaya iyi
InoteveraGaidhi rinotevera
Subword Tokenization
Mutauro AI