شينشيلا حساب التدريب الأمثل

Hoffmann and colleagues trained over 400 language models, from about 70 million to over 16 billion parameters, on roughly 5 billion to 500 billion tokens. They used three approaches to estimate how model size and training tokens should share a compute budget. First, they fixed model sizes, varied training horizons, smoothed and interpolated training curves, and selected the lowest loss for a given FLOP count. Second, they constructed IsoFLOP profiles by varying model size at fixed compute budgets and fitting the loss minimum on each profile. Third, they fit a parametric loss function over parameter count and token count, using Huber loss and L-BFGS with a grid of initializations. The three methods produced similar near-equal scaling estimates in the studied regime. Chinchilla itself was a 70B model trained on 1.4T tokens using MassiveText, with a slightly different subset distribution from Gopher.