Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Exclude biases and normalization scales from weight decay
You apply AdamW to a model with LayerNorm or BatchNorm by passing model.parameters() with one weight-decay value.
hardpractitioner heuristic - When a Transformer diverges early, check normalization placement and warmup
A Transformer trained from scratch shows loss spikes or NaNs in the first few hundred steps at a learning rate that works later in training.
hardempirical finding - When BERT-style fine-tuning fails on some seeds, train longer with a smaller learning rate
Fine-tuning an encoder on a small classification set succeeds on some seeds and collapses to majority-class predictions on others.
hardempirical finding - Watch for vocabulary drift in embedding-based models and adapt the encoder only when it pays
A complaint-topic clustering or classification model built on general-purpose sentence embeddings degrades after a new loyalty programme and new fare products introduce unfamiliar jargon.
hardempirical finding - Use label smoothing knowing it changes calibration and distillation
You turn on label smoothing for a classifier that will later serve as the teacher for a smaller distilled model.
hardempirical finding - Interpolate fine-tuned and original weights to keep robustness a fine-tune would erase
Fine-tuning a strong pretrained model lifts accuracy on your validation split, but the model is used on inputs that look different from the fine-tuning data.
hardempirical finding - Centre or whiten sentence embeddings before nearest-neighbour search and compare with the raw vectors
Scores between unrelated texts from a BERT-based encoder are all high, and every query seems to retrieve similar neighbours.
hardpractitioner heuristic - Count how often each item shows up in top-k lists to catch hub documents in embedding search
In a vector search over support articles, the same handful of articles appears in the top 10 for many unrelated queries.
hardpractitioner heuristic - Count selection data as part of the budget when you only own a few dozen labels
A few-shot text classifier is reported at a score tuned on prompts and learning rates using a large extra validation set, though production will have only 60 labels.
hardempirical finding - Keep per-item annotator votes and test soft labels before collapsing them to a majority
Three annotators label each tone or intent example, the pipeline keeps the majority label, and a third of the items were split 2 to 1.
hardempirical finding - Test on sequences longer than any seen in training before assuming the position scheme extends
A model trained on sessions of up to 128 events will see sessions of 400 events in production.
hardempirical finding