Neural networks, deep learning, NLP · insight-1456
Run a partial-input baseline to expose annotation artifacts before celebrating a text score
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A text-pair or template-labelled dataset scores high and you suspect the label writers left stylistic traces.
Action
Train deliberately handicapped baselines (one segment only, length only, first tokens only) and compare with the full model; if the gap is small, the benchmark partly measures artifacts.
Reason
Gururangan et al. found that a classifier seeing only the hypothesis predicts the NLI label far above chance, about 67 percent on SNLI and 53 percent on MultiNLI, and that negation words correlate with particular labels.
Boundary
Pair classification, templated notes, generated or crowd-written text.
Symptom or risk
The model may use one half of the input, or cues such as negation words or length, rather than the relationship you meant to teach.
Exceptions
Single-input tasks, where length-only and first-tokens baselines play the same role.
Trade-offs
A few extra training runs.
Pitfalls
Reporting only the full model; fixing one artifact and not re-running the baselines.
Lessons and concepts
Claims and their sources
Gururangan et al. report that a hypothesis-only classifier reaches about 67 percent on SNLI and about 53 percent on MultiNLI, and that negation and vague wording correlate with specific labels.