Critical Batch Size

A batch is the set of training examples used to form an update. Increasing its size averages over more examples and can reduce gradient noise. With enough parallel hardware, a larger batch may also reduce how many sequential updates are needed to achieve a chosen performance level. But those improvements do not continue proportionally forever. Critical batch size marks a characteristic transition in this tradeoff. In the empirical model of McCandlish and colleagues, batches well below the critical scale are relatively efficient per example. Far above it, adding examples to each update brings diminishing reductions in the number of updates. This is a smooth change in efficiency, not a universal hard limit beyond which training cannot work. The research connects the transition to gradient noise scale and finds that it can shift as training progresses. Define a common target before comparing runs. For an invented example, batch 256 reaches that target in 4,000 updates, processing 1,024,000 examples. Batch 512 takes 2,200 updates but processes 1,126,400 examples. It saves 45% of the updates while using 10% more examples. That could be an attractive elapsed-time tradeoff, but only measured update times reveal the runtime benefit. These two observations alone do not identify a precise critical batch size. Sweep several global batch sizes and tune relevant optimizer settings fairly. Record examples or tokens, updates, elapsed time, and validation performance at the same target. Keep track of whether a learning-rate schedule is expressed in updates or processed data, since changing the batch changes that relationship. Memory capacity, communication, and kernel efficiency impose additional practical limits that the statistical concept alone does not describe.