AI Glossary Term

What does “Pretraining” mean?

Definition

Initial large-scale model training on broad data before downstream adaptation.

Browse the full AI Glossary Explore all free AI guides

How Pretraining works

Example

A language model is first trained for weeks on a vast mix of text, learning grammar, facts, and styles by predicting the next token. Only afterwards is it shaped into a chat assistant through further training.

Why it matters

Pretraining is the most expensive stage and gives a model most of its general abilities. Choices about the data shape its strengths and biases.

Common questions

What data is used?

Usually large collections of web pages, books, code, and other text, sometimes plus images and audio, filtered for quality.

Is a pre-trained model ready to chat?

Not usually. It tends to continue text rather than follow instructions until it is further tuned.

Who can afford to pretrain?

Mostly large labs and well-funded groups, since it needs large compute budgets.

Post-trainingTraining steps applied after pretraining, such as instruction tuning, preference optimization, and safety tuning.Big DataVery large and complex datasets that require scalable storage and processing techniques.Continual LearningTraining approaches that let a model keep learning from new data without forgetting prior knowledge.Data AugmentationTechniques that create modified training examples to improve model generalization.Fine-TuningContinuing training on domain-specific data to adapt a pre-trained model to a specific task.Foundation ModelA large pre-trained model that can be adapted to many downstream tasks.

Learn more in our free guides

ELECTRA Pretraining Continued Pretraining vs Fine-Tuning CLAP: Contrastive Language-Audio Pretraining

Test yourself

ELECTRA Pretraining Quiz Continued Pretraining vs Fine-Tuning Quiz CLAP: Contrastive Language-Audio Pretraining Quiz