Neural networks, deep learning, NLP · insight-0473
Profile the input pipeline when GPU utilization stays low
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- ML engineer
- Industries
- retail, manufacturing
Summary: situation, action, reason, boundary
Situation
A GPU training job runs slowly and GPU utilization hovers well below full while the CPU is busy.
Action
Time one epoch of the data loader alone, then increase
num_workers, enablepin_memory, cache decoded data or move heavy augmentation to the GPU, and re-measure step time.Reason
The training step can only run as fast as batches arrive; host-to-GPU copies are faster from pinned memory.
Boundary
Image, audio and text pipelines with decoding or heavy augmentation.
Symptom or risk
Data loading and augmentation, not the model, set the pace; buying faster GPUs or shrinking the model changes nothing.
Exceptions
Small in-memory tabular data rarely bottlenecks on loading.
Trade-offs
More workers use more RAM and can duplicate dataset copies per process.
Pitfalls
Measuring throughput on the first epoch only, when caches are cold.
Lessons and concepts
Claims and their sources
PyTorch documents that host-to-GPU copies are much faster from pinned memory and that DataLoader pin_memory=True puts fetched tensors in pinned memory.