AI Glossary Term

What does “Model Quantization” mean?

Definition

Reducing numeric precision of model weights to decrease memory and inference cost.

Browse the full AI GlossaryExplore all free AI guides

How Model Quantization works

Diagram for Model Quantization comparing Full precision (16 or 32 bits per weight, More memory, Slower) with Quantized (8 or 4 bits per weight, Much smaller, Faster, small quality loss).Full precision16 or 32 bits perweightMore memorySlowerQuantized8 or 4 bits perweightMuch smallerFaster, smallquality loss

Example

A language model that needs 28 GB of memory at 16-bit precision can fit in roughly a quarter of that at 4-bit precision, letting it run on a consumer graphics card with a small drop in quality.

Why it matters

Quantization reduces memory and cost, which makes bigger models usable on smaller hardware. Quality losses should be tested, not assumed.

Common questions

Does quantization hurt accuracy?

Usually slightly, and more at very low precision. Good methods keep the loss small for many tasks.

When is quantization applied?

Often after training, but it can also be built into training.

Is it the same as Quantization (the glossary term)?

Yes, the two terms describe the same idea of lowering numeric precision for model weights.

Related terms

QuantizationConverting model weights to lower precision formats such as 8-bit or 4-bit.QLoRAA fine-tuning technique that combines 4-bit weight quantization with LoRA adapters to reduce memory needs.Flash AttentionAn optimized attention algorithm that reduces memory use and speeds up transformer training and inference.Memory (Agent Memory)Stored context an AI agent uses across steps or sessions to improve continuity.AI AgentA software system that can observe, reason, and take actions to achieve a goal, often using tools and memory.AI SafetyA field focused on reducing harmful behavior, failures, and misuse risks in AI systems.

Learn more in our free guides

AI Models ExplainedModel CollapseModel LifecycleModel Context Protocol

Test yourself

AI Models Explained QuizModel Collapse QuizModel Lifecycle Quiz

See also

Model DriftMultimodal ModelModel CardNamed Entity Recognition (NER)Mixture of Experts (MoE)Natural Language Processing (NLP)Neural NetworkMachine Learning (ML)