What does Token mean? Definition | AI Understanding

Definition

A chunk of text processed by language models, such as a word piece or symbol.

Related terms

AI systems that produce new content such as text, images, audio, video, or code.

Large Language Model (LLM)

A language model trained on massive text corpora to generate and analyze text.

Multimodal Model

A model that can process or generate multiple data types such as text, image, and audio.

A language-agnostic tokenizer that learns subword units directly from raw text without pre-splitting on whitespace.

A benchmark of grade-school math word problems used to evaluate step-by-step reasoning in language models.

The process of splitting text into tokens for model input.

Learn more in our free guides

SentencePiece Tokenization WordPiece Tokenization Tokenizer-Free Byte-Level Models Speculative Streaming and Multi-Token Prediction

See also

Temperature System Prompt Tool Use Synthetic Data Top-k Sampling Supervised Learning Top-p (Nucleus) Sampling Sparse Model

Browse the full AI Glossary Explore all free AI guides