Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Exclude biases and normalization scales from weight decay
You apply AdamW to a model with LayerNorm or BatchNorm by passing model.parameters() with one weight-decay value.
hardpractitioner heuristic - When a Transformer diverges early, check normalization placement and warmup
A Transformer trained from scratch shows loss spikes or NaNs in the first few hundred steps at a learning rate that works later in training.
hardempirical finding - Treat Monte Carlo dropout uncertainty as a signal to validate, not a calibrated interval
Stakeholders want an uncertainty band around a neural demand forecast and you consider keeping dropout on at inference and averaging many passes.
hardpractitioner heuristic - When BERT-style fine-tuning fails on some seeds, train longer with a smaller learning rate
Fine-tuning an encoder on a small classification set succeeds on some seeds and collapses to majority-class predictions on others.
hardempirical finding - Watch for vocabulary drift in embedding-based models and adapt the encoder only when it pays
A complaint-topic clustering or classification model built on general-purpose sentence embeddings degrades after a new loyalty programme and new fare products introduce unfamiliar jargon.
hardempirical finding - Scale the loss and the schedule correctly when you accumulate gradients
You emulate a batch of 256 with 8 micro-batches of 32 because the full batch does not fit in memory.
hardpractitioner heuristic - Use label smoothing knowing it changes calibration and distillation
You turn on label smoothing for a classifier that will later serve as the teacher for a smaller distilled model.
hardempirical finding - Start residual blocks as the identity by zero-initialising their last scale
A deep residual network trains with a very large batch, or its loss is erratic during the first hundred steps even after warmup.
hardempirical finding - Measure steps to target at several batch sizes before paying for bigger batches
A team wants to move a training job from 4 to 32 accelerators and assumes the run will finish about 8 times sooner.
hardempirical finding - Estimate the gradient noise scale to see when a larger batch has stopped helping
Training is stable at your current batch size and you are deciding whether to double it, but a full sweep is too expensive.
hardempirical finding - Re-check calibration on shifted data, because a temperature fitted in distribution may not travel
A temperature-scaled network is deployed to a new region, season or device mix and its probabilities feed a threshold chosen on the old validation split.
hardempirical finding - Interpolate fine-tuned and original weights to keep robustness a fine-tune would erase
Fine-tuning a strong pretrained model lifts accuracy on your validation split, but the model is used on inputs that look different from the fine-tuning data.
hardempirical finding - Include Adam's epsilon in the search space before ranking optimizers
A comparison of SGD with momentum and Adam is run with default settings, and Adam loses by a point on a tabular-style network.
hardempirical finding - Embed numerical features before the MLP when a tabular network trails gradient boosting
A plain MLP on raw standardised numerical features of a tabular problem (fare, days to departure, load factor) underperforms a gradient-boosted model by a clear margin.
hardempirical finding - Do not stop at the first rise in validation loss without checking a longer horizon
An over-parameterised network's validation error rises in mid-training, then, with patience set to five epochs, training halts and a smaller model is chosen instead.
hardempirical finding - Centre or whiten sentence embeddings before nearest-neighbour search and compare with the raw vectors
Scores between unrelated texts from a BERT-based encoder are all high, and every query seems to retrieve similar neighbours.
hardpractitioner heuristic - Count how often each item shows up in top-k lists to catch hub documents in embedding search
In a vector search over support articles, the same handful of articles appears in the top 10 for many unrelated queries.
hardpractitioner heuristic - Count selection data as part of the budget when you only own a few dozen labels
A few-shot text classifier is reported at a score tuned on prompts and learning rates using a large extra validation set, though production will have only 60 labels.
hardempirical finding - Keep per-item annotator votes and test soft labels before collapsing them to a majority
Three annotators label each tone or intent example, the pipeline keeps the majority label, and a third of the items were split 2 to 1.
hardempirical finding - Reweight to the production class prior when labelling effort was split evenly across intents
A labelling budget was split equally over six intents so each has the same count, while production traffic has one dominant intent and several rare ones.
harddocumented fact - Test on sequences longer than any seen in training before assuming the position scheme extends
A model trained on sessions of up to 128 events will see sessions of 400 events in production.
hardempirical finding - Detach the hidden state between truncated-BPTT chunks and treat chunk length as the learnable dependency horizon
You train an RNN on one long stream by carrying the hidden state across consecutive chunks.
harddocumented fact - Do not assume a CNN uses its full theoretical receptive field
You size a CNN on paper so its theoretical receptive field covers the whole image or sensor window and conclude the context is covered.
hardempirical finding