Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Assert prediction and target shapes right before every loss call
A regression network returns predictions of shape (N, 1) while the target tensor is (N,), for example predicted load factor per flight-day.
easydocumented fact - Count parameters by hand before training and compare with your number of rows
You size a multilayer perceptron for a tabular problem such as predicting final bookings from 20 booking-curve features.
easydocumented fact - Feed raw logits to CrossEntropyLoss, never softmax outputs
A multi-class model ends in nn.Softmax and its output goes straight into nn.CrossEntropyLoss.
easydocumented fact - Use BCEWithLogitsLoss instead of a sigmoid followed by BCELoss
A binary model, such as a no-show or fraud classifier, ends in a sigmoid layer and is trained with BCELoss.
easydocumented fact - Check that the first loss value equals what chance predicts
You start a new classifier training run, for example a 5-class complaint router.
easypractitioner heuristic - Make sure BatchNorm uses running statistics when you evaluate and serve
Validation accuracy differs between two scoring scripts, or a model scores the same image differently in a batch of one and in a full batch.
easydocumented fact - Turn dropout off at inference unless you are deliberately sampling
A deployed demand model returns slightly different predictions for the same request, and nobody changed the code.
easydocumented fact - Overfit a tiny batch before launching full training
A new model or data pipeline is ready and you are about to start a multi-hour training run.
easypractitioner heuristic - Compute input normalization statistics on the training split only
You standardize tabular inputs for a neural net with a mean and standard deviation computed on the full dataset before splitting.
easydocumented fact - Beat a tuned gradient-boosting baseline before shipping a neural net on tabular data
A team proposes a deep model for a tabular problem, such as predicting final bookings per flight from booking-curve, fare and calendar features.
easyempirical finding - Compute CNN feature-map sizes on paper before writing the classifier head
You stack convolution and pooling layers for an image task and hard-code the input size of the first linear layer.
easydocumented fact - Reproduce the pretrained model's preprocessing exactly before fine-tuning
You load a pretrained image or text backbone and feed it data prepared by your own pipeline.
easypractitioner heuristic - Score embeddings with the similarity function the model was trained for
You build semantic search over fare-rule documents and pick cosine similarity by habit for an embedding model you have not read about.
easydocumented fact - Run the first training step on the largest batch and longest sequence you will see
A text model trains with dynamic padding, and batch memory depends on the longest document in each batch.
easypractitioner heuristic - Measure GPU memory with the framework's allocator counters, not nvidia-smi alone
nvidia-smi shows a GPU almost full after training steps, and the team concludes the model cannot fit a larger batch.
easydocumented fact - Profile the input pipeline when GPU utilization stays low
A GPU training job runs slowly and GPU utilization hovers well below full while the CPU is busy.
easypractitioner heuristic - Call the scheduler after the optimizer and decide whether it steps per batch or per epoch
A schedule that should warm up over 500 steps is configured through a scheduler object, and the training loop was written by combining examples from different library versions.
easydocumented fact - Checkpoint the optimizer, scheduler and step counter, not only the weights
A long training job is preempted and you resume it from a file that holds only the model weights.
easydocumented fact - Look at a labelled grid of augmented samples before you train on them
You add random crops, flips, jitter or noise to a training pipeline and plan to judge the result only by validation accuracy.
easypractitioner heuristic - Write a run manifest that lets a stranger repeat the experiment
A model that scored well three weeks ago cannot be rebuilt: the notebook, the library versions and the data extract have all changed.
easypractitioner heuristic - Register submodules in ModuleList or ModuleDict, never in a plain Python list
A model builds a variable number of layers in a loop and stores them in a Python list or dict attribute.
easydocumented fact - Train once with the inputs zeroed to prove the model uses its features
A network reaches a decent score, but its loss curve looks similar to what a constant prediction would give.
easypractitioner heuristic - Get a model that overfits the full training set before you add regularisation
A new network underperforms and you immediately add dropout, weight decay and augmentation to it all at once.
easypractitioner heuristic - Shuffle the training loader explicitly, because the default does not
A DataLoader is built from rows exported sorted by route, date or label, and training loss is jagged with a repeating pattern of the same length as the epoch.
easydocumented fact - Normalise Unicode text to one form before tokenising, hashing or deduplicating
Turkish messages arrive from several apps and keyboards, and the same letter with a cedilla or breve is sometimes one precomposed character and sometimes a base letter plus a combining mark.
easydocumented fact - Left-pad batched prompts for decoder-only generation and confirm batched outputs match single runs
You batch prompts of different lengths for a decoder-only model, using the tokenizer's default right padding.
easydocumented fact - Set padding_idx on embeddings and still mask the pool, or padding length leaks into the features
A text model uses nn.Embedding with pad id 0 and mean-pools token vectors over the padded length.
easydocumented fact