Neural networks, deep learning, NLP · insight-1425
Quote an error bar that reflects the size of a small test set before comparing networks
- Evidence type
- empirical finding
- Difficulty
- easy
- Roles
- Data scientist, Data analyst
- Industries
- healthcare
Summary: situation, action, reason, boundary
Situation
You compare two neural classifiers on a hold-out of 120 rows and one is ahead by four points.
Action
Report a bootstrap or exact interval over test rows for each model and for their paired difference, use repeated cross-validation when the data are tiny, and state the number of test examples next to every score.
Reason
With about a hundred samples, cross-validated estimates carry error bars on the order of ten percentage points that the spread across folds does not capture.
Boundary
Small clinical, industrial or survey datasets and any hold-out below a few thousand rows.
Symptom or risk
With so few examples, the uncertainty of the measured accuracy is of the same order as the gap, so the ranking is mostly noise, and the winner may be the one you tuned on that set.
Exceptions
Very large differences, for example 20 points on a few hundred rows, may be real.
Trade-offs
Honest intervals will often say "not distinguishable", which is a decision-relevant result.
Pitfalls
Using the fold-to-fold standard deviation as the error bar; it understates the uncertainty.
Example
Synthetic: 100 test rows, accuracy 0.80 vs 0.84. Binomial standard errors are about 0.04 each, so the standard error of the unpaired difference is about 0.054, larger than the four-point gap.
Lessons and concepts
Claims and their sources
Varoquaux (arXiv preprint; journal venue not confirmed here) reports that cross-validation error bars are often underestimated and that for typical neuroimaging sample sizes of about 100 the uncertainty is on the order of plus or minus 10 percent, which the spread across folds does not capture.