Neural networks, deep learning, NLP · insight-0406
Use BCEWithLogitsLoss instead of a sigmoid followed by BCELoss
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- banking
Summary: situation, action, reason, boundary
Situation
A binary model, such as a no-show or fraud classifier, ends in a sigmoid layer and is trained with
BCELoss.Action
Drop the final sigmoid, train on logits with
BCEWithLogitsLoss, and apply the sigmoid only at prediction time. Use itspos_weightargument if you need to reweight the positive class.Reason
The fused loss computes the same quantity with the log-sum-exp trick, so it stays finite where the two-step version loses precision.
Boundary
Binary and multi-label classification in PyTorch, especially under mixed precision where range is tighter.
Symptom or risk
For confident wrong predictions the sigmoid saturates to exactly 0 or 1 in floating point; BCELoss then clamps the log (loss capped, flat gradient) and a hand-written log gives inf or NaN.
Exceptions
None for training; post-hoc calibration still needs probabilities, which you compute from logits.
Trade-offs
Serving code must remember to apply the sigmoid; a wrapper module avoids divergence.
Pitfalls
Keeping the sigmoid in the model and also using the logits loss, which applies the sigmoid twice.
Lessons and concepts
Claims and their sources
BCEWithLogitsLoss combines a sigmoid and binary cross-entropy and is more numerically stable than a separate sigmoid plus BCELoss because it uses the log-sum-exp trick.