What is Pruning?

An attention pattern where each token attends only to a fixed-size window of nearby tokens to reduce compute.

An empirical relationship showing how performance improves with model size, data, or compute.

A smaller model trained to imitate a larger model's behavior while using less compute at inference.

The core training algorithm that updates model weights by propagating prediction errors backward through the network.

The processing resources required to train and run models, often measured in FLOPS or GPU hours.

A vector showing how much each parameter should change to reduce loss.