Estimating GPU Memory Needed for LLMs

Model weights are one part of GPU memory. A first-order estimate is parameter count multiplied by bytes per stored parameter, but actual artifacts may use mixed precision, quantization scales, metadata, or multiple weight copies. The runtime may also reserve memory for kernels and workspaces. Leave headroom rather than planning to consume every advertised byte. Training adds gradients and optimizer state to the weights. Full fine-tuning generally needs more memory than inference because it must retain information for backward computation and updates. Activations saved between layers depend on architecture, batch size, sequence length, and precision. Gradient checkpointing can trade extra compute for lower activation memory; sharding distributes some state across devices. LoRA and other parameter-efficient methods reduce trainable state but still require activations and model weights. Autoregressive inference also uses a key-value cache so previously processed context can be reused during generation. For standard attention, an approximate cache size is two times the number of cached layers, batch size, cached tokens, key-value heads, head dimension, and bytes per cache element. The factor of two is for keys and values. Grouped-query attention can use fewer KV heads than query heads. Sliding-window attention may cap the cache for some layers; cache implementations and quantization add further details. Memory can therefore grow with both context length and concurrent sequences even after weights fit. Larger batches may improve throughput but increase cache and activation use. For long prompts, the KV cache can rival or exceed the weight memory. Multi-GPU sharding, offloading, quantization, and reduced batch size are possible tradeoffs, each affecting speed or complexity. Use formulas to narrow hardware choices, then profile the exact model, runtime, precision, context, and concurrency. Check peak memory during prefill and generation, fragmentation, and reserved memory. A rule of thumb is not a guarantee: architecture, kernels, serving engine, and cache policy all matter.