What is Tokenization?

The maximum amount of input tokens a language model can process at once.

A subword tokenization algorithm that merges the most frequent character pairs into reusable tokens.

The component of a model that transforms input into latent representations.

A model that can process or generate multiple data types such as text, image, and audio.

Technology that converts text in images or scans into machine-readable text.

A language-agnostic tokenizer that learns subword units directly from raw text without pre-splitting on whitespace.