Neural networks, deep learning, NLP · insight-0424

Overfit a tiny batch before launching full training

Situation
A new model or data pipeline is ready and you are about to start a multi-hour training run.
Action
Train on a fixed batch of 2 to 32 examples with regularization off until training loss is near zero and predictions match labels exactly. Only then train on the full data.
Reason
Networks can memorize even random labels, so failing to memorize a tiny batch points to a bug, not to insufficient capacity or data.
Boundary
Every new architecture, loss, label encoding or data loader.

Symptom or risk

Bugs in labels, loss wiring, masking or the optimizer step let training run but learn nothing useful; you discover it hours later from flat curves.

Heavily augmented or stochastic objectives need augmentation and dropout disabled for the test.

A few minutes per change; passing it proves the pipeline can learn, not that it generalizes.

Shuffling or augmenting the tiny batch so the target keeps changing; leaving weight decay on and blaming the model.

Karpathy's recipe recommends overfitting a single small batch to the lowest achievable loss as a bug check before full training.
A Recipe for Training Neural Networks
Zhang et al. show standard image CNNs trained with SGD easily fit randomly labelled training data.
Understanding deep learning requires rethinking generalization