Skip to content
ML Academy

project-031 · Deep learning · insurance

Few labels, foreign domain: frozen ImageNet features vs a small CNN on EuroSAT land cover

A (hypothetical) property and crop insurer wants to tag land cover around insured locations from satellite tiles to prioritise exposure reviews, but can pay for only a few hundred labelled tiles. With 20, 100 or 500 labelled tiles per class, does a frozen ImageNet backbone with a linear head beat a small CNN trained from scratch and a colour-statistics baseline, and at what labelling budget does the ranking change?

Licence
MIT (dataset); Copernicus Sentinel data terms apply to the underlying imagery
Reference run
none

At a glance

  • Data

    EuroSAT (RGB version) (public) · 27,000 RGB tiles, 64×64 pixels, 10 land-use and land-cover classes; 94.7 MB zip (a 13-band multispectral version is 2.1 GB)

    Details on data

  • Baseline

    Colour statistics plus logistic regression: per-channel mean, standard deviation, 10th/50th/90th percentiles and 16-bin histograms (about 60 features).

    Details on baseline

  • Validation

    Freeze a stratified 20% test set (5,400 tiles) and a 50-per-class validation pool. From the remaining tiles, draw labelled training sets of 20, 100 and 500 per class,…

    Details on validation

  • Metric

    Macro-F1 (primary), accuracy, per-class recall and the confusion matrix at each label budget; mean ± sd over label draws;

    Details on metric

Success criterion

  • A learning-curve table: macro-F1 (mean ± sd over 3 random label draws) for each method at 20, 100 and 500 labels per class, on one fixed test set.
  • Measured compute cost per method on your hardware (minutes for feature extraction and training).
  • A recommendation naming the cheapest method within one sd of the best at the labelling budget the business can afford, with the domain-shift caveat stated.

Context

Day 40 (choose a transfer-learning strategy under data and compute constraints), Day 36 and the Day 56 neural project. Docs-only in the academy: the project needs PyTorch, torchvision, a 44.7 MB weight download and a 94.7 MB dataset download, so the academy does not run it. CPU is enough for the core: torchvision lists ResNet-18 at 1.81 GFLOPS per 224×224 image, so one feature pass over 27,000 upsampled tiles is about 49 TFLOP of work, roughly minutes to tens of minutes on a laptop CPU depending on its throughput (estimate; measure yours and cache the features). Feeding the native 64×64 tiles costs about 12 times less, but the features change; that is an ablation, not a free speed-up. A GPU is useful only for the optional full fine-tuning. The scientific question is real: feature transferability falls as the target task moves away from the source task (Yosinski et al., 2014), and top-down satellite tiles are far from ImageNet photos.

Data

Dataset
EuroSAT (RGB version) (public)
Licence
MIT (dataset); Copernicus Sentinel data terms apply to the underlying imagery
Size
27,000 RGB tiles, 64×64 pixels, 10 land-use and land-cover classes; 94.7 MB zip (a 13-band multispectral version is 2.1 GB)
Verification
identity verified

Access

EuroSAT_RGB.zip (94.7 MB) from Zenodo record 7711810, linked from the GitHub README; also eurosat/rgb in TensorFlow Datasets. No authentication. Fallback: the deprecated mirror listed in the README. If every source is blocked, the project stays specified; do not substitute another dataset's results.

Grain

One row = one 64×64 Sentinel-2 tile with one class label taken from its folder name.

ColumnMeaning
file_pathPath inside the unzipped archive; the parent folder name is the class.
classOne of 10 land-use / land-cover classes; read the exact names from the folders and record them in the data card.
pixels64×64×3 uint8 RGB values derived from Sentinel-2 bands.
splitNo official split (TFDS exposes a single 'train' split of 27,000); you create and freeze one.

EDA and data-quality rules

  • Count tiles per class and record the folder names; check every file decodes to 64×64×3.

  • Exact and near-duplicate detection (perceptual hash) because neighbouring tiles can look alike; remove cross-split near-duplicates and report how many.

  • A grid of 16 tiles per class; note visually similar classes before modelling so error analysis has hypotheses.

  • Per-channel mean and standard deviation from the training split only, for the from-scratch CNN.

Baseline

Colour statistics plus logistic regression: per-channel mean, standard deviation, 10th/50th/90th percentiles and 16-bin histograms (about 60 features). Cheap, explainable and surprisingly informative for water, forest and built-up areas. Majority class is the floor.

Features

Three representations compared at each label budget: (a) hand-made colour statistics; (b) the 512-dimensional penultimate-layer output of ResNet-18 with IMAGENET1K_V1 weights, using the documented inference transform (resize 256, centre crop 224, ImageNet mean and std), extracted once in evaluation mode and cached to disk; (c) features learned by a small CNN trained from scratch at 64×64 with flips and 90-degree rotations, which are label-preserving for top-down imagery.

Candidate methods

  • Frozen ResNet-18 features + multinomial logistic regression (linear probe)

    Uses no gradients through the backbone; the cheapest transfer strategy and the one that suits tiny label budgets.

  • Small CNN from scratch (three conv blocks of 32, 64, 128 channels, global average pooling, dense 10)

    Learns satellite-specific filters; should improve as labels grow and is the fair 'no transfer' comparison.

  • Partial fine-tuning (unfreeze ResNet-18 layer4, small learning rate)

    Optional homework; adapts high-level features to the new domain. CPU-feasible at 20-100 labels per class; a GPU helps at 500.

Validation design

Freeze a stratified 20% test set (5,400 tiles) and a 50-per-class validation pool. From the remaining tiles, draw labelled training sets of 20, 100 and 500 per class, three random draws each, and train every method on the same draws. Regularisation strength and early stopping use the validation pool only. Read the test set once per final configuration. Caveat to state: tiles are geo-referenced, and a random split can place neighbouring tiles on both sides, which inflates scores; a spatially blocked split needs the georeferenced multispectral release (optional homework).

Leakage checks

Cross-split near-duplicates; normalisation statistics from training tiles only; the backbone in evaluation mode during extraction (in training mode, batch normalisation would use batch statistics, including test batches); no test tiles in the regularisation search.

Bias and representativeness

Geographic coverage: the dataset name points to European Sentinel-2 imagery; confirm the countries covered in the paper before using the model elsewhere, because land-cover appearance changes with climate, farming practice and building style. Class balance in your draws is forced; real portfolios are not balanced.

Robustness checks

Brightness and contrast changes (season and illumination), JPEG re-compression, 90-degree rotations for the linear probe (ImageNet features are not rotation-invariant), and the 64 vs 224 input-size ablation. Report spread over the three label draws.

Technical metrics

Macro-F1 (primary), accuracy, per-class recall and the confusion matrix at each label budget; mean ± sd over label draws; the learning curve of macro-F1 against labels per class on a log scale.

Business interpretation

Share of tiles auto-tagged at 95% precision for the classes that matter to the hypothetical insurer (for example built-up and water classes; use the actual folder names). Combine a hypothetical labelling cost per tile with the measured compute minutes to state the cost of each point on the learning curve.

Error slices

  • Confusions among vegetation and crop classes, which share colour and texture at 64×64.

  • Each label budget: where the ranking of methods flips.

  • Brightness quantiles of tiles (very dark or hazy tiles).

  • Rotated tiles for the linear probe against the from-scratch CNN trained with rotations.

Implementation steps

  • Before the studio (homework from the previous day): download data and weights, verify checksums, extract and cache ResNet-18 features; record minutes taken.

  • Studio 0-20 min: freeze the test set and validation pool, draw the label budgets, run the colour baseline.

  • Studio 20-50 min: linear probe on cached features for every budget and draw; fill the learning-curve table.

  • Studio 50-80 min: small CNN at 100 labels per class with shape assertions and a one-batch overfit check; compare.

  • Studio 80-90 min: decision-log entry on which strategy fits which budget.

  • Homework: CNN at all budgets; optional layer4 fine-tuning; 64 vs 224 ablation; robustness and slices; one test read; one-page report.

Reproduction

Pinned torch and torchvision versions in your own environment (not the academy's), weights selected by enum (ResNet18_Weights.IMAGENET1K_V1), cached feature arrays saved with a SHA-256 and the exact transform recorded, label-draw seeds saved to a file, hardware and minutes logged per step. One command reproduces the learning-curve table from the cached features; a second rebuilds the cache. No academy reference implementation.

Limitations

  • ImageNet-to-satellite is a large domain gap; a good linear-probe result here does not show that ImageNet features suit other sensors or resolutions.

  • Random splits of geo-referenced tiles are optimistic; the spatially blocked check is optional and needs the 2.1 GB release.

  • RGB only; the 13-band release carries information (for example near-infrared) that this project ignores.

  • The torchvision page does not state a licence for the pretrained weights; check before commercial use.

Production and monitoring

Pin the backbone version and preprocessing as a contract: a weight or transform change silently changes every feature. New imagery must match the training resolution and band processing; monitor the embedding distribution (for example mean distance to class centroids) for season and sensor drift, keep a human review queue for low-confidence tiles, and document the licence status of the weights and the Copernicus terms in the model card.

Interview questions

  • You have 50 labelled images per class and no GPU. Which transfer strategy do you choose and why?

  • Why must the backbone be in evaluation mode when you extract features, and what goes wrong otherwise?

  • Your linear probe on ImageNet features beats a from-scratch CNN at 20 labels per class but loses at 500. Explain the crossover.

Postmortem and alternatives

Record where the curves crossed and whether the 64 vs 224 ablation changed the conclusion. If the from-scratch CNN overtakes the probe at modest budgets, the domain gap is large enough that labelling more tiles beats importing features. Alternatives: a backbone pretrained on remote-sensing imagery, self-supervised pretraining on unlabelled tiles, or the multispectral bands with a small CNN.

Rubric4 criteria

CriterionMeetsExceeds
Experimental designFixed test set, three label budgets with three draws each, the same draws for every method.Adds the 64 vs 224 ablation or a spatially blocked check and explains how it changes the conclusion.
Transfer mechanicsCorrect transforms, evaluation mode, cached features with a checksum; linear probe tuned on validation only.Includes partial fine-tuning with a justified learning rate and compares its cost and benefit.
Compute and cost reportingMeasured minutes per method on stated hardware.Combines compute and a stated hypothetical labelling cost into a cost-per-macro-F1 comparison.
DecisionNames a strategy per budget with the domain-shift and geographic-coverage caveats.Specifies the monitoring signal and the evidence that would trigger switching strategy.

References6