Neural networks, deep learning, NLP · insight-0450
Do not present attention heatmaps as the reason for a prediction
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, Applied AI, ML engineer
- Industries
- banking
Summary: situation, action, reason, boundary
Situation
A stakeholder asks why the model flagged a loan application or complaint, and the team shows an attention heatmap over the input words.
Action
Treat attention as a hypothesis. Check it with perturbation tests (remove or mask the highlighted tokens and measure the change) and compare with attribution methods; report the explanation method and its limits.
Reason
Jain and Wallace found attention often does not track feature importance; Wiegreffe and Pinter argue the answer depends on the definition of explanation and propose tests, which is a reason to test rather than assume.
Boundary
Any Transformer or attention-RNN explanation shown to users, auditors or regulators.
Symptom or risk
Attention weights can disagree with gradient-based importance and can often be changed without changing the prediction, so the explanation may be wrong and still persuasive.
Exceptions
Exploratory debugging, where attention maps are a quick, clearly labelled hint.
Trade-offs
Perturbation tests cost extra inference and need careful baselines.
Pitfalls
Averaging attention over heads and layers and presenting it as one importance score.
Lessons and concepts
Claims and their sources
Jain and Wallace report that attention weights often do not correlate with gradient-based feature importance and that different attention distributions can give equivalent predictions.
Wiegreffe and Pinter argue that whether attention is explanation depends on the definition used and propose tests such as a uniform-weights baseline.