AI Glossary Term

What is Token?

Definition

A chunk of text processed by language models, such as a word piece or symbol.

Related terms

Generative AI
AI systems that produce new content such as text, images, audio, video, or code.
Large Language Model (LLM)
A language model trained on massive text corpora to generate and analyze text.
Multimodal Model
A model that can process or generate multiple data types such as text, image, and audio.
SentencePiece
A language-agnostic tokenizer that learns subword units directly from raw text without pre-splitting on whitespace.
GSM8K
A benchmark of grade-school math word problems used to evaluate step-by-step reasoning in language models.
Tokenization
The process of splitting text into tokens for model input.

Learn more in our free guides

SentencePiece TokenizationWordPiece TokenizationTokenizer-Free Byte-Level ModelsSpeculative Streaming and Multi-Token Prediction
Browse the full AI GlossaryExplore all free AI guides