AI Glossary Term

What does “SentencePiece” mean?

Definition

A language-agnostic tokenizer that learns subword units directly from raw text without pre-splitting on whitespace.

Browse the full AI GlossaryExplore all free AI guides

How SentencePiece works

Diagram for SentencePiece, steps in order: Raw text, spaces included; then Learn pieces from the data; then Treat space as a symbol; then Works for any language.Raw text, spaces includedLearn pieces from the dataTreat space as a symbolWorks for any language

Example

For languages such as Japanese or Thai that do not separate words with spaces, SentencePiece learns its pieces directly from the text without needing a word-splitting step first.

Why it matters

A language-agnostic tokenizer simplifies multilingual models. It is used in several well-known models.

Common questions

Who created it?

Taku Kudo and John Richardson at Google, in 2018.

Which algorithms does it support?

BPE and a unigram language-model method.

SentencePiece vs BPE?

BPE is an algorithm; SentencePiece is a toolkit that can use BPE or other algorithms on raw text.

Related terms

CLIPA multimodal model architecture that learns shared representations between text and images.Continual LearningTraining approaches that let a model keep learning from new data without forgetting prior knowledge.In-Context LearningA model's ability to follow patterns from examples provided directly in the prompt.Intent ClassificationPredicting the user's purpose from a text query to route it correctly.Large Language Model (LLM)A language model trained on massive text corpora to generate and analyze text.TokenA chunk of text processed by language models, such as a word piece or symbol.

Learn more in our free guides

SentencePiece Tokenization

Test yourself

SentencePiece Tokenization Quiz

See also

BPE (Byte Pair Encoding)ANN (Approximate Nearest Neighbor)Sliding Window AttentionHNSWALiBiRerankerRoPE (Rotary Positional Embeddings)Hybrid Search