Neural networks, deep learning, NLP · insight-0402
Test that examples in a batch cannot influence each other's predictions
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You wrote custom reshaping, pooling or attention code with
view,transposeor reductions over several axes.Action
Run a dependency test: take the loss of one example only, call backward, and assert that input gradients are non-zero for that example and exactly zero for all others. Also compare one example scored alone and inside a batch.
Reason
Backpropagation exposes which inputs a given output depends on, so a few lines of autograd turn a silent design bug into a failing test.
Boundary
Custom layers, sequence packing, masking logic and any model where batch elements should be independent.
Symptom or risk
A wrong axis mixes information across the batch dimension. The network usually still trains, so the bug survives until predictions change with batch composition in production.
Exceptions
Batch normalization in training mode legitimately couples examples; run the test in eval mode.
Trade-offs
Cheap to run once per architecture change; it adds a unit test that needs a deterministic tiny model.
Pitfalls
Testing with batch size 1 cannot reveal cross-batch mixing; use at least 3 distinct examples.
Lessons and concepts
Claims and their sources
Karpathy's training recipe describes using backprop to chart dependencies as a way to catch code that mixes information across the batch dimension.