Neural networks, deep learning, NLP · insight-0412
Monitor the share of dead ReLU units after a learning-rate increase
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- manufacturing
Summary: situation, action, reason, boundary
Situation
You raised the learning rate to speed training, and accuracy then stalls below where it was at the lower rate.
Action
Log, per layer, the fraction of units that are zero for an entire validation batch. If it climbs, lower the learning rate or add warmup, and consider a leaky or parametric ReLU.
Reason
ReLU has zero gradient for negative inputs, so a unit whose pre-activation is negative for every input stops receiving updates and cannot recover.
Boundary
ReLU MLPs and CNNs trained from scratch with aggressive learning rates.
Symptom or risk
A large update can push many ReLU units into a region where they output zero for every input; they get no gradient and never recover, shrinking effective capacity.
Exceptions
Transformer blocks with GELU and normalization are less exposed; a few dead units are harmless.
Trade-offs
Leaky variants reduce dead units but change sparsity and may need retuning.
Pitfalls
Measuring dead units on one training batch, where noise makes a unit look dead or alive.
Lessons and concepts
Claims and their sources
He et al. introduced the parametric ReLU, which learns a slope for negative inputs.