LLMs, GenAI, RAG, agents, evaluation · insight-0552

Budget about 16 bytes per parameter before activations for full fine-tuning with Adam

Situation
A team asks whether full fine-tuning of a 7-billion-parameter model fits on one 80 GB GPU.
Action
Estimate model states as roughly 16 bytes per parameter for mixed-precision Adam (weights, gradients, master weights and two moments), add activations, and compare with LoRA or QLoRA memory before choosing a method or hardware.
Reason
The ZeRO paper itemises mixed-precision Adam model states at 16 bytes per parameter, so a 7B model needs about 112 GB before activations.
Boundary
Planning full fine-tuning, choosing between full fine-tuning and PEFT, sizing clusters.

Symptom or risk

They count only the 16-bit weights (about 14 GB), start a run and hit out-of-memory errors, losing days to trial and error.

Sharding (ZeRO, FSDP), 8-bit optimizers or other optimizers change the total; recompute for your setup.

Memory-saving techniques add communication or compute overhead.

Ignoring activation memory, which grows with batch size and sequence length.

Rajbhandari et al. state that mixed-precision training with Adam holds 16 bytes of model states per parameter (fp16 weights and gradients plus fp32 master weights, momentum and variance).
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models