Neural networks, deep learning, NLP · insight-1462
Keep the 1/sqrt(d_k) factor when you write attention by hand, and log how peaked the weights are
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You hand-write scaled dot-product attention for a small sequence model and drop the division as cosmetic.
Action
Divide scores by sqrt(d_k) or call the library function, and log the mean largest attention weight at initialisation; a value near 1.0 means saturation.
Reason
Vaswani et al. note that with independent unit-variance components a dot product has variance d_k, and large magnitudes push softmax into regions with very small gradients.
Boundary
Custom attention layers and ports between frameworks.
Symptom or risk
Score variance grows with the key dimension, so softmax weights become near one-hot, gradients through them shrink, and training stalls or depends on luck.
Exceptions
Libraries that fold the scale into the query projection.
Trade-offs
None meaningful.
Pitfalls
Scaling by d_k instead of its square root; scaling after the softmax.
Example
Simulated locally with random normal vectors, d_k = 64 and 8 keys: unscaled score standard deviation about 8 and mean largest weight 0.88; scaled, standard deviation about 1 and mean largest weight 0.36.
Lessons and concepts
Claims and their sources
Vaswani et al. scale dot products by 1/sqrt(d_k) because for unit-variance independent components the dot product has variance d_k and large values push the softmax into regions with extremely small gradients.