Neural networks, deep learning, NLP · insight-1471
Do not assume a CNN uses its full theoretical receptive field
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer, Data scientist
- Industries
- manufacturing, healthcare
Summary: situation, action, reason, boundary
Situation
You size a CNN on paper so its theoretical receptive field covers the whole image or sensor window and conclude the context is covered.
Action
Measure the effective field with the gradient of a central output with respect to the input on a trained model, and add dilation, downsampling, skip connections or attention where it is too small.
Reason
Luo et al. report that the effective receptive field follows a Gaussian distribution and covers only a fraction of the theoretical one.
Boundary
Deep CNNs for images, audio and sensor series.
Symptom or risk
The effective receptive field is smaller and concentrated near the centre, so large objects or long-range cues are used less than the paper calculation suggests.
Exceptions
Tasks that are local by nature.
Trade-offs
More architectural complexity and compute.
Pitfalls
Reading a gradient map from an untrained network; using one image as evidence.
Lessons and concepts
Claims and their sources
Luo et al. (arXiv preprint) report that the effective receptive field follows a Gaussian distribution and occupies only a fraction of the theoretical receptive field.