Neural networks, deep learning, NLP · insight-1412
Put the task losses on one scale before adding them in a multi-task network
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A shared network predicts a revenue amount in euros and a cancellation probability, and the total loss is the plain sum of a squared error and a cross-entropy.
Action
Standardise regression targets, log each task's loss separately, and choose weights by validation metric or by learned uncertainty weighting; then confirm each head beats its single-task baseline.
Reason
Multi-task performance depends strongly on relative loss weighting, and weighting by learned task uncertainty was proposed so tasks with different scales can be learned together without hand-tuned weights.
Boundary
Shared encoders with heads of different types or units.
Symptom or risk
The squared error on a large-scale target dominates every gradient, the classification head barely trains, and the result changes when someone rescales a target from euros to thousands of euros.
Exceptions
Heads with identical loss type and scale on similar targets can start with equal weights.
Trade-offs
Learned weights add parameters and can drift toward the easiest task.
Pitfalls
Reading the summed training loss as one number; a falling total can hide a stalled head.
Lessons and concepts
Claims and their sources
Kendall et al. weight each task's loss by its homoscedastic uncertainty so regression and classification targets of different scales can be learned jointly, because multi-task performance depends on the relative loss weights.