What is Tokenization? Definition | AI Understanding

Ibisobanuro

Inzira yo kugabana inyandiko mubimenyetso byerekana icyitegererezo.

Amagambo afitanye isano

Umubare ntarengwa winjiza ibimenyetso byururimi rushobora gutunganyirizwa icyarimwe.

BPE (Encoding ya Byte)

Ijambo ryibanze tokenisation algorithm ihuza inyuguti ebyiri zikunze kugaragara mubimenyetso byongeye gukoreshwa.

Ibigize icyitegererezo gihindura ibyinjijwe muburyo bwihishe.

Icyitegererezo gishobora gutunganya cyangwa kubyara amakuru menshi nkinyandiko, ishusho, n'amajwi.

OCR (Kumenyekanisha neza)

Tekinoroji ihindura inyandiko mumashusho cyangwa scan mumashini isomwa.

Ururimi-agnostic tokenizer yiga subword ibice biturutse kumyandiko mbisi itabanje gutandukana kumwanya wera.

Wige byinshi mubuyobozi bwacu

SentencePiece Tokenization WordPiece Tokenization Tokenization Subword Tokenization

See also

Token Tool Use Temperature Top-k Sampling System Prompt Top-p (Nucleus) Sampling Synthetic Data Transfer Learning

Reba Amagambo yuzuye ya AI Shakisha ibyerekezo byose bya AI