What is Token? Definition | AI Understanding

Ibisobanuro

Igice kinini cyinyandiko yatunganijwe nicyitegererezo cyururimi, nkijambo igice cyangwa ikimenyetso.

Amagambo afitanye isano

Sisitemu ya AI itanga ibintu bishya nk'inyandiko, amashusho, amajwi, amashusho, cyangwa code.

Ururimi runini (LLM)

Ururimi rwicyitegererezo rwahuguwe kumyandiko minini corpora kubyara no gusesengura inyandiko.

Icyitegererezo gishobora gutunganya cyangwa kubyara amakuru menshi nkinyandiko, ishusho, n'amajwi.

Ururimi-agnostic tokenizer yiga subword ibice biturutse kumyandiko mbisi itabanje gutandukana kumwanya wera.

Ibipimo byamanota-shuri imibare yamagambo ibibazo bikoreshwa mugusuzuma intambwe ku yindi ibitekerezo mubitekerezo byururimi.

Inzira yo kugabana inyandiko mubimenyetso byerekana icyitegererezo.

Wige byinshi mubuyobozi bwacu

SentencePiece Tokenization WordPiece Tokenization Tokenizer-Free Byte-Level Models Speculative Streaming and Multi-Token Prediction

See also

Temperature System Prompt Tool Use Synthetic Data Top-k Sampling Supervised Learning Top-p (Nucleus) Sampling Sparse Model

Reba Amagambo yuzuye ya AI Shakisha ibyerekezo byose bya AI