Neural networks, deep learning, NLP · insight-0459
Report per-class recall and macro averages for long-tail intent classifiers
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Data analyst, Applied AI
- Industries
- banking
Summary: situation, action, reason, boundary
Situation
A banking chatbot intent model reports 94% accuracy over 60 intents, and most traffic falls into five intents.
Action
Report a per-class table with support, precision and recall, plus macro F1 and precision-recall curves for the costly classes. Set alerts on the recall of named critical intents, not on overall accuracy.
Reason
Under imbalance, metrics that average over examples hide minority-class performance; precision-recall views reflect minority performance better than ROC.
Boundary
Multi-class text classification with skewed label distributions.
Symptom or risk
Accuracy and micro averages are dominated by frequent intents, so complete failure on rare but costly intents, such as fraud reports, stays invisible.
Exceptions
When all classes have similar costs and frequencies, overall accuracy is a reasonable summary.
Trade-offs
Per-class metrics on tiny classes are noisy; show counts and intervals.
Pitfalls
Optimizing macro F1 blindly, which can sacrifice a high-volume class that matters commercially.
Lessons and concepts
Claims and their sources
Saito and Rehmsmeier show precision-recall plots are more informative than ROC plots for imbalanced binary classifiers.