Neural networks, deep learning, NLP · insight-1422

Watch the update-to-weight ratio per layer to size a learning rate

Situation
Training is slow or erratic and you cannot tell whether the learning rate is too small for some layers or too large for others.
Action
Every few hundred steps log, per parameter tensor, the norm of the update divided by the norm of the weights. A rough target near one in a thousand is a starting point; investigate tensors far above or below the rest.
Reason
The ratio measures how much each tensor actually changes per step, so it is comparable across layers whose raw gradient scales differ widely.
Boundary
Fully connected and convolutional networks trained with SGD or Adam.

Symptom or risk

A global learning rate that suits the output layer may barely move early layers, or may churn them, and the loss curve alone does not say which.

Pretrained layers you deliberately fine-tune slowly; the target there is smaller.

Extra logging cost; sample steps instead of logging every one.

Treating the rule of thumb as a law; some architectures work well far from it.