AI Glossary Term

What does “Token” mean?

Definition

A chunk of text processed by language models, such as a word piece or symbol.

Browse the full AI Glossary Explore all free AI guides

How Token works

Example

The word 'unbelievable' might be split into three tokens, while a short common word like 'the' is one. Providers count tokens to limit input and to price usage.

Why it matters

Tokens determine cost, speed, and how much fits in the context window. Different languages can need different token counts for the same meaning.

Common questions

How many words is a token?

In English, a token averages roughly three-quarters of a word, but it varies by tokenizer and language.

Why not use whole words?

Pieces handle rare words and new spellings by combining familiar parts.

Are spaces and punctuation tokens?

They are often included in or alongside tokens.

Generative AIAI systems that produce new content such as text, images, audio, video, or code.Large Language Model (LLM)A language model trained on massive text corpora to generate and analyze text.Multimodal ModelA model that can process or generate multiple data types such as text, image, and audio.SentencePieceA language-agnostic tokenizer that learns subword units directly from raw text without pre-splitting on whitespace.GSM8KA benchmark of grade-school math word problems used to evaluate step-by-step reasoning in language models.TokenizationThe process of splitting text into tokens for model input.

Learn more in our free guides

SentencePiece Tokenization WordPiece Tokenization Tokenizer-Free Byte-Level Models Speculative Streaming and Multi-Token Prediction

Test yourself

SentencePiece Tokenization Quiz WordPiece Tokenization Quiz Tokenizer-Free Byte-Level Models Quiz