Neural networks, deep learning, NLP · insight-0471
Measure GPU memory with the framework's allocator counters, not nvidia-smi alone
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
nvidia-smi shows a GPU almost full after training steps, and the team concludes the model cannot fit a larger batch.
Action
Log
torch.cuda.max_memory_allocated()andmax_memory_reserved()per step, reset peaks between experiments, and size batches from the allocated peak plus a safety margin.Reason
PyTorch documents that unused memory held by its caching allocator still appears as used in nvidia-smi, and exposes separate counters for allocated and reserved memory.
Boundary
PyTorch training and inference on CUDA GPUs.
Symptom or risk
PyTorch's caching allocator keeps freed memory reserved, so nvidia-smi overstates what tensors use and capacity decisions are based on the wrong number.
Exceptions
Other processes or frameworks on the same GPU are visible only in nvidia-smi; check both when sharing.
Trade-offs
Fragmentation can make reserved memory matter; watch both counters.
Pitfalls
Calling empty_cache() to fix out-of-memory errors, which only releases cached blocks to other processes.
Lessons and concepts
Claims and their sources
PyTorch's CUDA notes state the caching allocator's unused memory still shows as used in nvidia-smi, and that memory_allocated and memory_reserved (with max variants) monitor tensor and cached memory.