Estimating LLM Training FLOPs

Training compute counts arithmetic operations, while FLOP/s measures how quickly a system performs them. Keep those units separate. For a conventional dense Transformer, a common first estimate is C ≈ 6ND, where N is the relevant parameter count and D is the total number of tokens processed during training. Repeated passes through the same tokens count again. State how embeddings and other parameters are counted so comparisons use the same convention.

The factor six approximates the main parameter-matrix work: roughly 2N operations per token in the forward pass and 4N in the backward pass. It is not an exact count of every operation. Attention, long sequences, architecture differences, and implementation choices can require a more detailed estimate. Applying the dense formula to a mixture-of-experts model using all stored parameters can be misleading because only some experts are active for each token.

Consider a constructed example with N = 10⁹ and D = 2 × 10¹⁰. Multiplying gives C ≈ 1.2 × 10²⁰ FLOPs. Suppose eight accelerators each have a relevant peak of 100 × 10¹² FLOP/s, with an assumed model FLOPs utilization of 0.5. Their effective model throughput is 4 × 10¹⁴ FLOP/s. Dividing compute by throughput gives 300,000 seconds, about 83.3 hours. Eight devices running that long represent about 667 accelerator-hours.

Measure a representative training pilot before committing to a schedule. Use the intended sequence length, batch size, precision, software, and device arrangement. Record both tokens per second and what the timing includes. Add explicit allowances for evaluation, checkpoints, interruptions, and experimentation when those activities fall outside the measurement. The arithmetic estimate alone does not show whether the model fits in memory.