Neural networks, deep learning, NLP · insight-0407
Pick softmax or independent sigmoids by asking whether labels can co-occur
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer, Applied AI
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
You classify flight disruption reports into causes (weather, crew, technical, ATC) and some reports cite several causes.
Action
Check the label data: if rows can carry several labels, use one logit per label with
BCEWithLogitsLossand a threshold per label. If exactly one label applies, use softmax withCrossEntropyLoss.Reason
Cross-entropy over a softmax models one class per example; binary cross-entropy per logit treats each label as a separate yes or no question.
Boundary
Tagging, document routing, symptom coding and any task where label sets, not single labels, are the target.
Symptom or risk
A softmax head forces the cause probabilities to sum to one, so a report with two true causes can never score both highly and recall on multi-cause reports collapses.
Exceptions
If co-occurrence is rare and the business needs one primary cause, a single-label model with a defined tie rule can be simpler.
Trade-offs
Multi-label needs per-label thresholds and per-label metrics, which means more tuning and reporting.
Pitfalls
Thresholding independent sigmoids at 0.5 for rare labels; tune thresholds per label on validation data.
Lessons and concepts
Claims and their sources
CrossEntropyLoss takes per-class logits and, with class-index targets, one target class per example, while BCEWithLogitsLoss scores each output independently and supports a per-class pos_weight.