What is Token? Definition | AI Understanding

Ma'anarsa

Rubutun da aka sarrafa ta tsarin harshe, kamar guntun kalma ko alama.

Sharuɗɗa masu alaƙa

AI mai ƙirƙira

Tsarin AI waɗanda ke samar da sabon abun ciki kamar rubutu, hotuna, sauti, bidiyo, ko lamba.

Babban Samfurin Harshe (LLM)

Samfurin harshe da aka horar akan babban haɗin gwiwar rubutu don samarwa da tantance rubutu.

Multimodal Model

Samfurin da zai iya sarrafawa ko samar da nau'ikan bayanai da yawa kamar rubutu, hoto, da sauti.

Alamar harshe-agnostic wanda ke koyan raka'a ƙananan kalmomi kai tsaye daga ɗan rubutu ba tare da an riga an raba kan farar fata ba.

Ma'auni na matsalolin kalmomin lissafi na aji-makaranta da aka yi amfani da su don kimanta tunanin mataki-mataki a cikin ƙirar harshe.

Tsarin rarraba rubutu zuwa alamu don shigarwar ƙira.

Ƙara koyo a cikin jagororinmu na kyauta

SentencePiece Tokenization WordPiece Tokenization Tokenizer-Free Byte-Level Models Speculative Streaming and Multi-Token Prediction

Duba kuma

Temperature System Prompt Tool Use Synthetic Data Top-k Sampling Supervised Learning Top-p (Nucleus) Sampling Sparse Model

Nemo cikakken AI ƙamus Bincika duk jagororin AI kyauta