Data-Constrained Scaling Laws

A training budget may permit more computation than the available corpus can use efficiently in a single pass. This happens in small domains, restricted language collections, and projects with a fixed set of approved documents. The question becomes how much to repeat the data, how large a model to train, and whether acquiring or improving data is more valuable than extending the run. Keep two token counts. Unique tokens describe the available corpus after the chosen processing and duplicate treatment. Processed tokens count every training exposure. With ten billion unique tokens and three complete passes, the run processes thirty billion tokens. The arithmetic tracks work, not thirty billion distinct pieces of evidence. A tokenizer or filtering change can also change the token count, so preserve the counting method when comparing experiments. The study Scaling Data-Constrained Language Models explicitly investigates this setting. Its experiments found that a limited amount of repetition could approach the loss achieved with additional unique data, while further repetition eventually offered diminishing value. The reported finding of small loss differences through roughly four epochs belongs to that study’s tested conditions; it is not a universal instruction to train every corpus four times. The work proposes a fitted scaling model that accounts for the reduced value of repeated data. Use that result to design a local experiment, not to skip one. Compare model sizes and repetition schedules under a stated compute budget, retain held-out data, and measure relevant task performance alongside loss. Keep evaluation examples out of training and check duplicate overlap. If later checkpoints improve training loss but worsen validation results, investigate memorization or mismatch rather than assuming more compute must help. Consider data quality, coverage, and permission to use candidate additions before enlarging the corpus.