Neural networks, deep learning, NLP · insight-0409
Check that the first loss value equals what chance predicts
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- telecom
Summary: situation, action, reason, boundary
Situation
You start a new classifier training run, for example a 5-class complaint router.
Action
Before training, compute the loss on one batch and compare it with , which is about 1.61 for 5 classes. For imbalanced binary tasks, initialize the output bias to the log-odds of the base rate.
Reason
A freshly initialized head gives near-uniform probabilities, so cross-entropy should start at the uniform value; a correct bias removes the first epochs spent learning the base rate.
Boundary
Any classification or regression setup; derive the analogous value for squared error from the target's mean and variance.
Symptom or risk
If the step-zero loss is far from the chance value, the head, labels or loss are wired wrong; you lose hours before the curves make that visible.
Exceptions
Pretrained heads, a bias initialized to the base rate and large initial logits change the expected starting loss; label smoothing does not move the uniform-prediction value.
Trade-offs
Bias initialization helps early training but can mask a label-encoding bug if you never check the uninitialized value.
Pitfalls
Reading a step-zero loss from a run that already took several optimizer steps.
Lessons and concepts
Claims and their sources
Karpathy's training recipe recommends verifying that the loss at initialization matches -log(1/n_classes) for a softmax classifier and initializing the final layer sensibly.