What is Pretraining?

Training steps applied after pretraining, such as instruction tuning, preference optimization, and safety tuning.

Very large and complex datasets that require scalable storage and processing techniques.

Training approaches that let a model keep learning from new data without forgetting prior knowledge.

Techniques that create modified training examples to improve model generalization.

Continuing training on domain-specific data to adapt a pre-trained model to a specific task.

A large pre-trained model that can be adapted to many downstream tasks.