Skip to content
ML Academy

project-029 · Deep learning · retail

Catalogue thumbnails: a small CNN vs pixel baselines on Fashion-MNIST

A (hypothetical) apparel retailer's catalogue team assigns one of ten garment types to every new product image. Can a small convolutional network pre-fill the category accurately enough that staff review only the low-confidence items, and how much does it gain over pixel-based classical models that are cheaper to maintain? Fashion-MNIST stands in for real catalogue images.

Licence
MIT
Reference run
none

At a glance

  • Data

    Fashion-MNIST (public) · 70,000 images (60,000 train, 10,000 test), 28×28 grayscale, 10 balanced classes; about 30 MB compressed

    Details on data

  • Baseline

    Majority class (10% accuracy, the floor) and multinomial logistic regression on the 784 scaled pixels.

    Details on baseline

  • Validation

    The official 10,000-image test set is held out and read once. A stratified 50,000/10,000 train/validation split of the official training set (fixed seed) is used for…

    Details on validation

  • Metric

    Accuracy, macro-F1, per-class recall, log loss and a reliability curve (modern networks can be over-confident, which matters when a confidence threshold routes work).

    Details on metric

Success criterion

  • A hand-written shape and parameter table for the CNN that matches the code (assertions pass; 421,642 parameters for the reference architecture below).
  • Accuracy and macro-F1 for the CNN and the best classical baseline on the official test set, read once, with a bootstrap 95% interval on the difference.
  • Auto-accept coverage at 98% precision, with the confidence threshold chosen on validation.
  • Recommend the CNN only if its coverage gain is outside the interval and its degradation under 2-pixel shifts is no worse than the baseline's; otherwise recommend the baseline.

Context

Days 34-36 and the Day 56 neural project. Images are the domain where the convolution's assumptions (local patterns, the same filter reused across positions) usually pay off, which makes this a useful contrast with the tabular project 028. Keep the comparison honest anyway: on small, centred, grayscale 28×28 images a dense network and even linear models are serious baselines, and real catalogue photos (backgrounds, poses, several items per image) are much harder. CPU only: the reference CNN needs roughly 25 million floating-point operations per image for a forward and backward pass (back-of-envelope), so an epoch over 60,000 images takes seconds to a few minutes on a laptop. Measure and record yours.

Data

Dataset
Fashion-MNIST (public)
Licence
MIT
Size
70,000 images (60,000 train, 10,000 test), 28×28 grayscale, 10 balanced classes; about 30 MB compressed
Verification
identity verified

Access

Four gzip files (MNIST format) linked from the repository README with MD5 checksums; also available through torchvision.datasets.FashionMNIST, TensorFlow Datasets and Hugging Face Datasets. No authentication; about 30 MB in total. Fallback if the network is blocked: run the pipeline end to end on scikit-learn digits (bundled) to test the code, and mark the Fashion-MNIST results as pending rather than substituting digits results.

Grain

One row = one product image with one garment-type label.

ColumnMeaning
image28×28 uint8 pixel intensities, 0-255.
labelInteger 0-9: T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, Ankle boot.
splitOfficial train (60,000) or test (10,000) file; you carve validation out of train.

EDA and data-quality rules

  • Verify the MD5 checksums from the README before use and record them in the data card.

  • Class counts per split; a grid of 25 random images per class and the mean image per class (the mean images show how centred the data are).

  • Exact-duplicate check by hashing pixel arrays, within train and across train and test; report counts and drop cross-split duplicates from training.

  • Confirm dtype and range, then scale by a constant (divide by 255) so no fitted statistic touches validation or test.

Baseline

Majority class (10% accuracy, the floor) and multinomial logistic regression on the 784 scaled pixels. The logistic model is the cheap production alternative the CNN has to beat by a margin that matters for review workload.

Features

Raw pixels divided by 255. Classical models see a flattened 784-vector; gradient boosting sees 50 PCA components fitted on the training split only. The CNN sees tensors of shape (N, 1, 28, 28). No augmentation in the main comparison; random shifts of up to 2 pixels are the regularisation experiment.

Candidate methods

  • Small CNN: conv3×3 (1→32, padding 1) → ReLU → max-pool 2 → conv3×3 (32→64, padding 1) → ReLU → max-pool 2 → dense 3136→128 → ReLU → dropout 0.3 → dense 128→10

    Shapes (N,1,28,28) → (N,32,28,28) → (N,32,14,14) → (N,64,14,14) → (N,64,7,7) → (N,3136) → (N,128) → (N,10). Parameters: 320 + 18,496 + 401,536 + 1,290 = 421,642, of which about 95% sit in the first dense layer, a useful talking point for global average pooling.

  • Dense MLP on flattened pixels (784→256→128→10, ReLU)

    Same family, no convolution. Separates 'a non-linear model helps' from 'convolution helps'; the gap should widen under pixel shifts if the convolutional bias matters.

  • HistGradientBoosting on 50 PCA components

    A strong classical non-linear baseline that many teams would try first; cheap to retrain.

Validation design

The official 10,000-image test set is held out and read once. A stratified 50,000/10,000 train/validation split of the official training set (fixed seed) is used for early stopping, threshold choice and model selection. CNN and MLP run with 3 seeds each. In the studio, use a 10,000/2,000 subset for speed and say so; the homework repeats on the full data.

Leakage checks

Cross-split duplicate check; constant scaling (or statistics fitted on train only); PCA fitted on train only; the confidence threshold chosen on validation; one read of the test set, logged with a timestamp.

Bias and representativeness

No people in the data. The relevant bias is representational: one retailer's studio style, centred single items on a plain background. Report per-class recall; the upper-body garment classes (shirt, T-shirt/top, pullover, coat) are the ones to watch in the confusion matrix. Check this on your own results rather than assuming it.

Robustness checks

Evaluate all models on test images shifted by 1 and 2 pixels, with Gaussian noise, and with brightness scaled by 0.7 and 1.3; plot accuracy against perturbation size per model. Report seed spread.

Technical metrics

Accuracy, macro-F1, per-class recall, log loss and a reliability curve (modern networks can be over-confident, which matters when a confidence threshold routes work). Mean ± sd over seeds; a bootstrap interval on the CNN-minus-baseline difference.

Business interpretation

Coverage at 98% precision: the share of images auto-categorised above the validation threshold. Translate into review hours with an explicitly hypothetical review time per image (for example 10 seconds), and state the cost of a misfiled item as an assumption you vary.

Error slices

  • Per class, with the confusion block among shirt, T-shirt/top, pullover and coat.

  • Low-confidence band (top-1 probability below the auto-accept threshold): what share of each class lands there.

  • Mean-intensity quantiles (very light or very dark items).

  • Shifted and noisy test images, by model.

Implementation steps

  • Studio 0-15 min: load, verify checksums, run the EDA rules, create the validation split and the studio subset.

  • Studio 15-35 min: majority and logistic-regression baselines; start the experiment log.

  • Studio 35-65 min: CNN. Fill in the shape and parameter table by hand first, then assert each shape in the forward pass and compare sum(p.numel()) with your count; check the initial loss is near ln 10 ≈ 2.303; overfit 64 images to near-zero loss; train 3 epochs on the subset.

  • Studio 65-90 min: compare on validation, plot the confusion matrix, write the decision-log entry.

  • Homework: full data with 3 seeds; dense MLP and PCA-boosting candidates; the shift-augmentation experiment as the regularisation study; robustness curves; one read of the test set; a one-page report.

Reproduction

Add a CPU build of PyTorch to your own project environment (the academy's python/ environment does not include it) and pin its version; a numpy im2col implementation is an acceptable but slower alternative. Seed numpy, random and torch, enable deterministic algorithms where available, and record the CPU model and wall time per epoch. One command, for example uv run python train.py --seed 0 --subset full, must regenerate the metrics file and figures. This project has no academy reference implementation.

Limitations

  • Centred 28×28 grayscale thumbnails are far easier than real product photos; the result does not transfer directly to a real catalogue.

  • The official test set has been used in a very large number of published comparisons; it is not new data for the field, only for your model.

  • Three seeds give a rough spread, not a precise interval.

  • Single-label images: no mechanism for multi-item photos or for new garment types outside the ten classes.

Production and monitoring

Preprocessing parity is the main risk: real images must be resized, centred and converted exactly as in training, so make the transform a tested function shared by training and serving. Route images below the confidence threshold to human review and monitor coverage, reviewer overrides by class and the rate of images the reviewers mark as 'none of the ten'. Retrain when override rates rise; publish a short model card with the training data's studio style as a stated limitation.

Interview questions

  • Compute the output shape and parameter count of the second convolution. Why does the first dense layer dominate the parameter count, and what would global average pooling change?

  • Your CNN beats logistic regression by three points on Fashion-MNIST. What would you need before claiming it will help on the real catalogue?

  • Why can a dense MLP and a CNN reach similar accuracy on centred images yet degrade differently when images shift by two pixels?

Postmortem and alternatives

Record which of three explanations your evidence supports: the CNN wins mainly through robustness to shifts; the dense MLP gets most of the gain, so non-linearity rather than convolution matters on centred images; or the gain is inside the seed noise. Alternatives for a real catalogue: a pretrained backbone with a linear head (project 031), global average pooling to cut parameters, or keeping logistic regression with a stricter review threshold if maintenance cost dominates.

Rubric4 criteria

CriterionMeetsExceeds
Shapes and parametersHand-written shape and parameter table matches the code; assertions in the forward pass; loss at init near ln 10.Explains where the parameters sit and shows the parameter and accuracy effect of replacing the dense head with global average pooling.
Fair comparisonSame split and metrics for all models, seeds reported, test read once, threshold chosen on validation.Includes the dense MLP to separate the effect of convolution from non-linearity and reports a bootstrap interval on the difference.
Robustness and error analysisPer-class recall, confusion matrix and the shift test for at least two models.Robustness curves for all candidates plus the augmentation experiment, with an explanation of which failures it fixes.
DecisionRecommendation with coverage at 98% precision and the hypothetical review-time assumption stated.Names what evidence from real catalogue images would be needed before deployment and how it would be collected.

References6