Core day 36 of 60 · Week 8 · Part II · orientation depth

CNNs

A plant engineer wants to classify 28 x 28 greyscale crops of weld seams as good or defective. A colleague proposes one convolutional layer: 8 filters of size 5 x 5, stride 2, padding 1. Before training, can you say what shape comes out, how many numbers the layer learns, and what one filter looks for?

By the end: For a convolution layer you compute the output shape (with padding, stride and dilation) and the parameter count by hand, explain what a filter detects, and say when a convolution fits non-image data and when it does not.

At a glance

One small filter, reused at every position, turns an image into maps of where each learned pattern appears; shapes and counts follow from five numbers.

Local filter
A k x k grid of weights looks at one small patch at a time.
Shared weights
The same filter at every position: 208 weights instead of a million.
Output size
floor((n + 2p - d(k - 1) - 1) / s) + 1 per axis; padding counts twice.
Equivariant
Shift the input and the feature map shifts; pooling adds invariance.
When it fits
Neighbouring values related and patterns meaning the same anywhere.

Recall · 5 min

No notes, one minute each: (1) the parameter count of a dense layer with 784 inputs and 10 outputs, biases included; (2) what a validation loss that rises while the training loss keeps falling tells you (Day 34); (3) the array shape of a batch of 32 greyscale 28 x 28 images. Today's idea in one line: reuse the same small weight pattern at every position.

Open due reviews

Concept · 10 min

A dense layer treats a 28 x 28 image as 784 unrelated numbers. Images, audio and sensor traces are different: nearby values belong together, and the same pattern (an edge, a scratch, a knock) can appear anywhere. A Convolution (filter, kernel) slides one small grid of learned weights over the input and records, at every position, how strongly its pattern appears.

Output size is arithmetic - kernel, padding, stride and dilation decide it, and the filter decides what lights up

Predict first

A 28 x 28 input, a 3 x 3 kernel, stride 1 and no padding. What is the output size?

Text description of the figure

A grid input with a sliding kernel. Controls for input size, kernel size, padding and stride update the output size and the number of weights, and a pattern filter highlights where the input matches it.

Four ideas behind a convolution layer Overview

Local filter
A small grid of weights, for example 3 x 3, multiplies one patch and sums.
Shared weights
The same filter slides over every position of the input.
Feature map
The output grid: how strongly the pattern appears at each position.
Pooling
A max or mean over a window or the whole map; no weights.

One filter slid over every position fills the feature map Illustration

Input: 0 = dark, 1 = bright. The highlighted box is the 3 x 3 window for output row 3, column 1.

Filter: right column minus left column. In the box: 3 products are non-zero, sum = 3.

10 - 3 + 1 = 8 positions per axis. Only 28 of 64 cells fire: + where dark turns bright, - where bright turns dark.

Same image, the transposed filter: it ignores the side edges and fires on the top and bottom edges instead.

Synthetic 10 x 10 image; cross-correlation without padding (no kernel flip), computed by the same code as the conv calculator. Blue = positive, red = negative response.

Each cell of the map is one window's sum of nine products. The same nine weights produce all 64 cells; the sign tells dark-to-bright from bright-to-dark.

Output size along one axis Formula

n_{out} = \left\lfloor \frac{n + 2p - d(k-1) - 1}{s} \right\rfloor + 1 \;=\; \left\lfloor \frac{28 + 2 - 4 - 1}{2} \right\rfloor + 1 = 13

$n, k$: input length and kernel size (28 and 5 for the engineer's layer)
$p$: zeros added on EACH side, hence 2p
$s$: stride: positions skipped between windows
$d$: dilation: gap between kernel taps (3 taps with d = 2 span 5 positions)
$\lfloor \cdot \rfloor$: floor: a window that does not fit is dropped

Shape trace of a small CNN for 28 x 28 seam crops (padding 1, stride 1 convs) Diagram

Seam crop28 x 28 x 1 greyscale
Conv, 8 filters 3x3out 28 x 28 x 8; 80 weights
- to Max pool 2 x 2
Dense 16 to 2good / defective; 34 weights
Max pool 2 x 2out 14 x 14 x 8; 0 weights
- to Conv, 16 filters
Global avg poolout 16: one mean per filter
- back to Dense 16 to 2
Conv, 16 filters3x3x8 each; out 14 x 14 x 16; 1,168 weights
Max pool 2 x 2out 7 x 7 x 16; 0 weights
- back to Global avg pool

Each box shows the tensor it outputs and the weights it learns: 1,282 in total, 0 in every pooling step. Conv weights: 8 x (9 + 1) = 80, then 16 x (3 x 3 x 8 + 1) = 1,168, because the second conv sees 8 input channels. Shapes from the output-size formula.

Learned numbers for the same 1,352 outputs from a 28 x 28 input Chart

Conv: 8 filters 5 x 5, stride 28 x 25 + 8208
Dense layer with 1,352 outputs784 x 1,352 + 1,3521,061,320

About 5,100 times fewer parameters, because one filter is reused at all 169 positions.

Move the pattern, the map moves with it; a global read-out does not Illustration

Input A: a plus-shaped pattern near the top-left corner.

The template filter scores 5 where it lines up exactly (ring), less on partial overlaps.

Input B: the same pattern, moved 3 rows down and 4 columns right.

The map moved by exactly (3, 4): equivariance. The peak is now at row 5, column 6 instead of 2, 2.

What survives the move
Read-out	A	B	Verdict
Feature map	peak 2,2	peak 5,6	moves with the input
Global max	5	5	identical: invariant
Global mean	0.31	0.31	identical: invariant
Peak cell after 2 x 2 max pool, 1-pixel shift	same in 42 of 72 shifts		invariant only sometimes

A 2 x 2 pool forgives a 1-pixel shift only when the peak stays inside the same window (58% of positions here): pooling gives local, partial invariance; a global read-out gives full invariance.

Synthetic 11 x 11 binary images and a hand-set template filter; maps from the same conv2d code as the calculator (no padding, stride 1). Every count in the table is computed.

Equivariance is a property of the feature map, invariance of what you read from it. Local pooling sits in between.

The output is a Feature map; the filter's weights are learned, so the network finds which small patterns reduce the loss.
Output size per axis follows from five numbers (input, kernel, padding, stride, dilation); the parameter count from kernel area, input channels and filters (Parameter sharing), never from the image size.
A convolution is shift-equivariant; Pooling adds invariance. Stacking layers widens the Receptive field.
The same arithmetic runs on one axis for vibration or ECG. With little labelled data, teams reuse pretrained filters (Transfer learning, Day 40).

Assumptions

Locality - neighbouring values in the array are related (pixels, adjacent time samples). A table with arbitrary column order breaks this.
Stationarity of patterns - the same pattern means the same thing wherever it occurs. If position matters (a label in a fixed corner, time of day), supply it explicitly.
Enough data for the task, or a pretrained model whose domain and preprocessing match yours.

Application and workplace decision · 10 min

The engineer's layer: input 28 x 28 x 1, 8 filters of 5 x 5, stride 2, padding 1. By hand, write:

(a) the output height and width;
(b) the learned weights including biases;
(c) the weights of a dense layer that produced the same number of outputs from the flattened input.

Then run the code, predicting first which output columns light up:

a from-scratch convolution (libraries compute cross-correlation; the kernel is not flipped);
the vertical-edge filter on a 6 x 6 image whose right half is bright;
the shape formula for four settings, and the parameter comparison.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np

def out_size(n, k, p=0, s=1, d=1):
    """Output length of one axis: floor((n + 2p - d(k-1) - 1) / s) + 1."""
    return (n + 2 * p - d * (k - 1) - 1) // s + 1

def conv2d(x, w, p=0, s=1, d=1):
    """Single-channel cross-correlation with padding, stride and dilation."""
    x = np.pad(x, p)
    k = w.shape[0]
    n_out = out_size(x.shape[0], k, 0, s, d)
    out = np.zeros((n_out, n_out))
    for i in range(n_out):
        for j in range(n_out):
            span = d * (k - 1) + 1
            patch = x[i * s : i * s + span : d, j * s : j * s + span : d]
            out[i, j] = (patch * w).sum()
    return out

img = np.zeros((6, 6)); img[:, 3:] = 1.0          # dark left half, bright right half
edge = np.array([[-1, 0, 1]] * 3, dtype=float)    # vertical-edge detector
resp = conv2d(img, edge)
print("response shape", resp.shape, "row 0:", resp[0])

x = np.random.default_rng(0).normal(size=(28, 28))
for p, s, d in [(0, 1, 1), (1, 2, 1), (2, 1, 1), (0, 1, 2)]:
    got = conv2d(x, np.ones((5, 5)), p, s, d).shape[0]
    assert got == out_size(28, 5, p, s, d)
    print(f"28x28, k=5, pad={p}, stride={s}, dilation={d} -> {got}x{got}")

filters, k, c_in = 8, 5, 1
side = out_size(28, k, 1, 2)
dense = 28 * 28 * (side * side * filters) + side * side * filters
print(f"conv params: {filters * (k * k * c_in) + filters}; dense with the same outputs: {dense:,}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 29 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The layer, defined3 minThe exact equation one output cell computes, every symbol explained, and the two assumptions written as properties of that equation.

One output cell, defined (stride 1, no padding) Formula

y[i,j] \;=\; b \,+\, \sum_{u=1}^{k}\sum_{v=1}^{k} w[u,v]\; x[i+u-1,\; j+v-1]

$y[i,j]$: one cell of the feature map: the response of the filter at position (i, j)
$w[u,v]$: the k x k learned weights - the SAME w for every (i, j), which is the sharing
$x[i+u-1, j+v-1]$: the input patch under the window; locality: nothing outside it enters y[i,j]
$b$: one learned bias per filter

A textbook convolution flips the kernel first; deep-learning libraries skip the flip (cross-correlation) because the weights are learned either way. Stride, padding and dilation only change which patches the window visits.

The equation, read as four properties Overview

Learned
The k x k weights w and one bias b per filter; nothing else.
Fixed before training
k, stride, padding, dilation and the number of filters: design choices.
Locality
y[i,j] reads one k x k patch; a pixel outside the window cannot change it.
Stationarity
The same w at every (i, j): a pattern is asserted to mean the same anywhere.

Cost follows from the same equation: one cell takes k x k x c_in multiply-adds; a full layer takes that times output positions times filters.

The sliding dot product, by hand4 minA 5 x 5 input pushed through a 3 x 3 filter cell by cell, the weight sharing counted, and the 6 x 6 edge example as a second pass.

The sliding dot product on a 5 x 5 input, cell by cell Sequence

The input
5 x 5, all 0 except column 3, which is 1: a one-pixel bright stripe (think a scratch on a part)
The filter
3 x 3 vertical-edge detector: every row is (-1, 0, 1); 9 weights, bias 0
Window at column 1
Covers columns 1-3. Each row: -1 x 0 + 0 x 0 + 1 x 1 = 1; three rows sum to +3
Window at column 2
Covers columns 2-4: the stripe sits under the 0 tap, so each row gives 0
Window at column 3
Covers columns 3-5: bright left, dark right: -1 x 1 + 0 + 1 x 0 = -1 per row, so -3
The feature map
3 x 3, every row (+3, 0, -3): position kept, sign = direction of the edge
Sharing, counted
9 weights reused at all 9 positions (81 multiplies); a dense 25-to-9 layer needs 225 weights

Verified in numpy: each output row is (3, 0, -3), and out_size(5, 3) = 3. Redo one window on paper before moving on.

A vertical-edge filter on a 6 x 6 image (left half 0, right half 1) Sequence

The filter
Each of its 3 rows is (-1, 0, 1): right column minus left column
Columns 1-3 (all dark)
Every pixel 0: response 0
Columns 2-4
Left column 0, right column 1, three rows: 3 x (0 x -1 + 1 x 1) = 3
Columns 3-5
Left column still dark, right column bright: response 3
Columns 4-6 (all bright)
1 x (-1) + 1 x 1 per row: response 0

Row of the feature map: (0, 3, 3, 0). The filter fires only where dark meets bright, and the 6-wide input gives 4 positions.

The output-size formula, derived3 minCount where windows can start and the full formula with padding, stride and dilation falls out; then four settings checked by the code.

Where the output-size formula comes from Sequence

Pad
p zeros on EACH side: the padded axis has n + 2p positions, indexed 0 to n + 2p - 1
Span
k taps with dilation d cover d(k - 1) + 1 positions (3 taps, d = 2: positions t, t+2, t+4)
Starts
Stride s means windows start at t = 0, s, 2s, ...
Last fit
The window at t ends at t + d(k - 1), which must be <= n + 2p - 1, so t_max = n + 2p - d(k - 1) - 1
Count
Starts 0..t_max in steps of s: floor(t_max / s) + 1. The floor drops a window that does not fit
Check
n = 28, k = 5, p = 1, s = 2: floor((28 + 2 - 4 - 1) / 2) + 1 = floor(12.5) + 1 = 13

This is the full formula from the size-eq figure, derived rather than memorised; the code asserts it for four settings.

The shape formula on 28 x 28 with a 5 x 5 kernel (checked by the code) Table

Padding	Stride	Dilation	Output
0	1	1	24 x 24
1	2	1	13 x 13
2	1	1	28 x 28 ('same' size)
0	1	2	20 x 20 (kernel spans 9)

Receptive field, derived and effective4 minHow far one output can see, three ways to widen it, the derivation, and why the usable field is smaller than the formula.

How far one output sees - three ways to stack 3-tap layers Illustration

Each stride-1 layer adds k - 1 = 2 samples: 7 after three layers.

Same 3 weights per layer, gaps between the taps: 15 samples. No stride, so the outputs stay dense (with padding they keep the input length).

Also 15 samples, but the output shrinks to 1 position: resolution is spent to buy context.

Stride-1 stacks grow linearly; doubling dilation or stride 2 grow exponentially (both give the same numbers). Log scale.

Index arithmetic on a 17-sample input, no padding; the highlighted cone is found by following every tap down the stack; it matches 1 + sum of (k - 1) x dilation x earlier strides. This is the theoretical field; trained networks weight its centre most.

Depth adds context linearly; dilation and stride add it exponentially. Stride (and pooling) pays with resolution; dilation keeps resolution but skips the samples between its taps.

Receptive field of stacked stride-1 layers Formula

r = 1 + \sum_{i} d_i (k_i - 1)

$k_i$: kernel size of layer i
$d_i$: dilation of layer i

With strides or pooling the growth is faster; each later layer's step is multiplied by the earlier strides.

Why the field grows by d(k - 1) per layer Sequence

One layer
An output reads k neighbouring inputs: r = k
Add a stride-1 layer
Its k taps read outputs one input apart, each already r wide; together they add d(k - 1) new inputs
Add stride or pooling below
Neighbouring taps now sit j inputs apart (j = product of earlier strides): the layer adds d(k - 1) x j
Check
Three 3 x 3 layers: 1 + 2 + 2 + 2 = 7. Dilation 1, 2, 4: 1 + 2 + 4 + 8 = 15

Induction on depth; the receptive-field demo above finds the same numbers by following every tap down the stack.

How far one output can see Table

Stack	Receptive field
One 3 x 3 layer	3
Two 3 x 3 layers	5
Three 3 x 3 layers	7
Two 5 x 5 layers	9
Three 3 x 3 layers, dilation 1, 2, 4	15

A pattern wider than the receptive field cannot be seen by one output, however the weights are shared.

The usable field is narrower: edge taps carry little (% of one output) Chart

input -4 (edge)1.2 %
-34.9 %
-212.3 %
-119.8 %
0 (centre)23.5 %
+119.8 %
+212.3 %
+34.9 %
+4 (edge)1.2 %

Four stacked 3-tap averaging layers (theoretical field 9), computed: each input's share of one output. The central 5 inputs carry 88%, the two edges 2.5%. Luo et al. (2017) found the effective field of real networks roughly Gaussian in the same way, so the pattern you need (a whole cycle, a whole bracket) must sit well inside the field, not at its edge.

Channels, 1 x 1 and depthwise-separable layersadvanced4 minWhere weights and multiply-adds really sit, and the cheap layer types that change the balance.

Parameters of a convolution layer with channels Formula

N = f \cdot (k_h k_w \, c_{in}) + f, \qquad 3 \times 3, \; 64 \to 128: \; 128 \cdot 576 + 128 = 73{,}856

$f$: number of filters = output channels
$k_h k_w c_{in}$: each filter spans all input channels
$+ f$: one bias per filter

Image size never appears. PyTorch's Conv2d stores the weight as (out channels, in channels, kh, kw).

Three conv layers, 64 to 128 channels: what each looks at and pays Comparison

Criterion	Standard 3 x 3	1 x 1 conv	Depthwise 3 x 3, then 1 x 1
Looks at	A 3 x 3 patch across all 64 channels at once	One position only: no spatial context	Each channel's own 3 x 3 patch (groups = 64)
Mixes channels	Yes, inside every filter	Yes: it only remixes channels per position	Only in the 1 x 1 step after it
Weights (with biases)	73,856	8,320; the cheap way to shrink or widen channels	640 + 8,320 = 8,960: about 8x fewer, same for multiply-adds

The standard layer does spatial filtering and channel mixing in one step; depthwise-separable splits them, which is where the saving comes from.

Where weights and compute go (64 to 128 channels, 56 x 56 map) Table

Layer	Weights	Multiply-adds per image
Standard 3 x 3 conv	73,856	231.2 million
Depthwise 3 x 3 + pointwise 1 x 1	8,960	27.5 million
1 x 1 conv, 64 to 16 (bottleneck)	1,040	3.2 million
Two 3 x 3 convs (64 to 64) vs one 5 x 5	73,728 vs 102,400	same 5 x 5 receptive field
Flatten 128 x 56 x 56, dense to 256	102,760,704	same as the weights
Global average pool, dense to 256	33,024	same as the weights

3 x 3 kernels, stride 1, padding 1; computed in Python, weights include biases. Convolutions hold the compute; a flatten-to-dense head holds the parameters. Two stacked 3 x 3 layers see as far as one 5 x 5 with fewer weights and an extra nonlinearity between them.

Pooling, equivariance and when not to convolve5 minWhat each pooling choice throws away, the equivariance-invariance distinction, the one-axis case, and the neighbours a CNN competes with.

From equivariant to invariant: what each stage keeps of position Comparison

Criterion	Convolution	Max pool 2 x 2	Average pool	Global pool
Computes	Shared weights at every position	Largest value in each window	Mean of each window	One max or mean per channel over the whole map
If the pattern moves	The map moves with it: equivariant, position kept	Unchanged only while the peak stays in its window	Same partial forgiveness; peaks are diluted	Identical value: invariant, all position lost
Typical use	Find the pattern wherever it is	Halve height and width between conv blocks	Smoother downsampling	Before the head; works for any input size

Equivariance belongs to the feature map, invariance to what you read from it. If position decides the label, global pooling removes exactly the information you need.

The same idea on one axis (vibration, ECG, electricity load) Diagram

Signal windown samples, 1 channel
Conv1d, kernel kn - k + 1 responses
Global max poolis the pattern anywhere?
Linear headclass score

The hands-on lab builds exactly this with a hand-set filter and shows what the pooled feature buys over raw samples.

When a convolution is the right tool Map

Unordered features

Column order is arbitrary; a filter has nothing local to find. Use trees or linear models.

Convolution fits

Images, audio, sensor traces, spectrograms.

Tabular data

Neither holds; tree ensembles are the default (Day 12).

Conv plus position

Local patterns whose meaning depends on where they are: add coordinates, avoid global pooling.

A CNN and its neighbours on images or signals Comparison

Criterion	CNN	MLP on raw pixels	Hand-built features + GBM	Attention over patches
Built-in assumption	Locality and the same pattern anywhere	None: every pixel has its own weight	Whatever you engineer	Any pair may interact; weak locality prior
Labelled data needed	Moderate; pretrained backbones help	Large	Small	Usually large, or pretrained at scale
Where it breaks	No locality, or position carries the label	A one-pixel shift changes every input	Patterns nobody thought to engineer	Cost grows with patches squared (Day 39)

The architecture is a bet about the data. Small labelled sets favour the strongest prior that is true for your data.

The knobs, and when a CNN fails on new dataadvanced4 minPyTorch Conv2d defaults and what each trades, then a decision flow that checks the pipeline and the split before blaming the architecture.

Conv2d knobs: PyTorch defaults and what each trades Table

Knob	Default	Trade-off
padding	0; 'same' allowed only with stride 1	Keeps border positions and map size vs a shrinking map
stride	1	Less compute and more context per layer vs lost resolution
dilation	1	Wider view with no extra weights vs skipped samples between taps
groups	1 (groups = in channels: depthwise)	Far fewer weights vs less mixing across channels
bias	True	Redundant right before batch norm, which subtracts the mean
init	uniform(-sqrt(k), sqrt(k)), k = groups / (c_in kh kw)	Scale shrinks with fan-in; many recipes re-initialise (He init for ReLU)

Defaults read from the PyTorch 2.14 Conv2d documentation page. Weight shape is (out channels, in channels / groups, kh, kw).

A CNN that validates well and fails on next week's images Diagram

Val good, new week bad97% random split, 81% next week
Same preprocessing?resize, crop, norm
- yes: to Split by seam or week?
no
Fix the pipelineassert one batch end to end
Split by seam or week?near-duplicates
- yes: to Few labels?
no
Re-split and re-measurethe old score was a leak
Few labels?
- no: to Train top blocks
yes
Frozen backbone + headplus a classical baseline
Train top blocksaugment only real variation

Check the cheap explanations (pipeline, split) before the architecture; a deeper network trained on the same leak only looks better.

What a staff-level answer addsadvanced2 minThe nuances that separate a textbook answer from an experienced one, and the common misreadings.

The textbook answer and what a staff-level answer adds Comparison

Criterion	Common reading	Staff-level reading
Weight sharing	A compression trick	A prior: a pattern means the same thing everywhere
Where the budget sits	The convolutions hold everything	Convs hold the compute; a flatten-to-dense head holds the parameters
Shift behaviour	'CNNs are translation invariant'	Equivariant; pooling adds invariance, stride breaks it for odd shifts
Seeing further	'A bigger kernel sees more'	Depth, dilation and stride usually widen the field more cheaply
Filter pictures	Show what the decision relies on	Suggest what a layer responds to; not evidence of reliance (Day 17)

Stride breaks invariance for any shift that is not a multiple of the stride: the sampled grid lands on different positions.

Memory anchor

One small filter, reused everywhere: the shape comes from five numbers, the weights from training.

Useful for

Checking a colleague's architecture sketch in minutes - shape and parameter arithmetic exposes impossible or wasteful layers before any training.
Choosing between a convolution, a dense layer and engineered features for data that has a spatial or temporal order.
Deciding whether a pretrained image backbone is a reasonable feature extractor for a small labelled set (defect photos on a line, retail shelf images, scanned insurance claim forms).

Not suitable for

Tabular rows with arbitrary column order - there is no locality for a filter to exploit; a gradient-boosted model or MLP is the first candidate.
Judging what a trained network relies on - filter pictures are suggestive, not evidence of the decision logic.
Choosing an architecture without a baseline; at orientation depth this lesson gives you the arithmetic, not a training recipe.

Traps

Off-by-one shapes that nothing catches

Padding is added on both sides and the formula takes a floor. A mistake of one position per layer compounds, and it surfaces either as a shape error at the flatten-to-dense step or, worse, silently when adaptive pooling hides it.

How to avoid it

Write the shape per layer on paper, then assert it with one dummy forward pass before training.

Using a convolution where there is no locality

A Conv1d across 200 tabular columns treats neighbouring columns as related. Reordering the columns changes the model's behaviour although no information changed, which is a sign the inductive bias does not match the data.

How to avoid it

Ask what "neighbour" means in your data; shuffle the column order once and see whether the score moves.

Senior reasoning

A senior treats a convolution as a bet about the data: that nearby values are related and that patterns repeat. They check that bet cheaply (shape arithmetic, a shuffled-column or shifted-input test), count where the parameters really sit (usually the flatten-to-dense layer), and prefer a frozen pretrained extractor plus a simple head when labels are scarce. They also ask for the classical baseline, which for many sensor tasks is hand-built features plus a gradient-boosted model.

Interview question

"Why do convolutional networks need far fewer parameters than dense ones on images, and when would you not use one?"

Outline your answer first.

Recall cards scheduled after completion

Write the output length of a convolution axis and the parameter count of a conv layer.
Is a convolution invariant or equivariant to shifting the input, and what adds invariance?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

A guide to convolution arithmetic for deep learning — Vincent Dumoulin, Francesco Visin, 2016core
Relates input size, kernel size, zero padding and strides to the output size of convolutions and pooling; the source for the shape arithmetic in the mechanism.
Conv2d (PyTorch documentation) — PyTorchcore
Gives the output-size formula including dilation and the weight shape (out_channels, in_channels/groups, kernel height, kernel width) behind the parameter count.
Understanding the Effective Receptive Field in Deep Convolutional Neural Networks — Wenjie Luo, Yujia Li, Raquel Urtasun, Richard Zemel, 2017optional
The effective receptive field is roughly Gaussian and covers only a fraction of the theoretical one (depth chapter on receptive fields).
Gradient-based learning applied to document recognition — Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, 1998optional
Convolutional networks with local receptive fields and shared weights trained end to end by gradient descent (LeNet-style document recognition).
How transferable are features in deep neural networks? — Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson, 2014optional
Early layers learn general features and later layers task-specific ones, which is the basis for reusing a pretrained feature extractor (developed on Day 40).

Optional sections

Deep dive · 10 min

Receptive field with stride. For a stack of layers, the receptive field grows by $(k_i - 1) \cdot j_i$ , where $j_i$ is the "jump" (the product of earlier strides) between neighbouring outputs of layer $i$ . Strides and pooling therefore widen the view faster than stride-1 layers, at the price of resolution.

Channels and 1 x 1 convolutions. A real layer mixes channels: each filter has shape $k \times k \times c_{in}$ and produces one map. A $1 \times 1$ convolution has $k = 1$ and so no spatial context; it recombines channels at each position, which is a cheap way to change channel counts.

Where the parameters go. A 3 x 3 convolution from 64 to 128 channels has $128 \cdot (9 \cdot 64) + 128 = 73,856$ weights. Flattening a $128 \times 56 \times 56$ map into 256 dense units needs about 102.8 million. In small image networks the flatten-to-dense layer, not the convolutions, holds most parameters; global average pooling before the head removes the dependence on the input resolution.

Non-image data. A 1-D convolution suits sensor windows, ECG, audio and short click sequences where the pattern can occur at any time. For a daily demand series, a convolution over the last 28 days is one of several window features you could build and has to beat lag features in a GBM (Day 24) and the strong baselines of Day 27 to justify itself.

Hands-on lab: A hand-built 1-D filter and global max pooling on shifted bursts (synthetic) · 15 min

Synthetic vibration windows: half contain a short damped burst at a random position, half are noise. A logistic regression on the raw samples cannot use position-dependent evidence; one filter matched to the burst, followed by global max pooling, turns the problem into a single feature. In a CNN the filter would be learned; here it is set by hand so you can see the mechanism. Change the burst amplitude from 2.0 to 1.0 and describe how accuracy moves, then explain why this is still not a claim about real bearings.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(36)
n, length, width = 600, 128, 9
# short damped "knock" shape (synthetic)
burst = np.hanning(width) * np.sin(np.linspace(0, 3 * np.pi, width))

def make(label, start=None):
    x = rng.normal(0, 0.5, length)
    if label:
        start = rng.integers(0, length - width) if start is None else start
        x[start:start + width] += 2.0 * burst
    return x

y = rng.integers(0, 2, n)
X = np.stack([make(l) for l in y])
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

raw = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
print("raw-signal logistic regression accuracy:", round(raw.score(Xte, yte), 3))

def conv_maxpool(X, w):
    windows = np.lib.stride_tricks.sliding_window_view(X, len(w), axis=1)  # (n, length-k+1, k)
    resp = windows @ w                                                     # (n, length-k+1)
    return resp.max(axis=1, keepdims=True), resp.shape

feat_tr, shape = conv_maxpool(Xtr, burst)
feat_te, _ = conv_maxpool(Xte, burst)
print("conv response length per signal:", shape[1], "= 128 - 9 + 1")
clf = LogisticRegression().fit(feat_tr, ytr)
print("conv + global max pool + logistic regression accuracy:", round(clf.score(feat_te, yte), 3))

a = make(1, start=30)
b = np.roll(a, 20)                                                         # same burst, shifted
ra = conv_maxpool(a[None], burst)[0][0, 0]
rb = conv_maxpool(b[None], burst)[0][0, 0]
print("max response, original vs shifted by 20:", round(float(ra), 2), round(float(rb), 2))

Interview practice · 10 min

A teammate at a bank wants a CNN for 150 tabular features about a personal loan (income, utilisation, tenure, ...). How do you respond?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-13 · depends on numpy, scikit-learn

1 Recall · 5 min

2 Concept · 10 min