Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Assert prediction and target shapes right before every loss call
A regression network returns predictions of shape (N, 1) while the target tensor is (N,), for example predicted load factor per flight-day.
easydocumented fact - Test that examples in a batch cannot influence each other's predictions
You wrote custom reshaping, pooling or attention code with view, transpose or reductions over several axes.
mediumpractitioner heuristic - Count parameters by hand before training and compare with your number of rows
You size a multilayer perceptron for a tabular problem such as predicting final bookings from 20 booking-curve features.
easydocumented fact - Check how many parameters your embedding tables hold before adding ID features
You add learned embeddings for high-cardinality identifiers such as customer, store or SKU IDs to a neural model.
mediumpractitioner heuristic - Feed raw logits to CrossEntropyLoss, never softmax outputs
A multi-class model ends in nn.Softmax and its output goes straight into nn.CrossEntropyLoss.
easydocumented fact - Use BCEWithLogitsLoss instead of a sigmoid followed by BCELoss
A binary model, such as a no-show or fraud classifier, ends in a sigmoid layer and is trained with BCELoss.
easydocumented fact - Pick softmax or independent sigmoids by asking whether labels can co-occur
You classify flight disruption reports into causes (weather, crew, technical, ATC) and some reports cite several causes.
mediumdocumented fact - Leave regression outputs linear and choose the loss for the statistic you need
A network predicts a non-negative quantity such as daily energy load or claim cost, and someone adds a ReLU on the output.
mediumpractitioner heuristic - Check that the first loss value equals what chance predicts
You start a new classifier training run, for example a 5-class complaint router.
easypractitioner heuristic - Calibrate a neural classifier before using its softmax output as a probability
A network's predicted probability will drive a cost-based decision, such as how much overbooking risk to accept or when to route a claim to manual review.
mediumempirical finding - Match weight initialization to the activation function
A deep MLP or CNN with ReLU activations trains very slowly, or a tanh network's loss does not move for many steps.
mediumdocumented fact - Monitor the share of dead ReLU units after a learning-rate increase
You raised the learning rate to speed training, and accuracy then stalls below where it was at the lower rate.
mediumpractitioner heuristic - Run a learning-rate range test before tuning anything else
You start training a new architecture or dataset and are about to grid-search learning rate together with other hyperparameters.
mediumpractitioner heuristic - Retune the learning rate whenever you change the batch size
You move training to a bigger GPU and raise the batch size from 64 to 512 to finish faster, keeping the old learning rate.
mediumempirical finding - Use Adam to iterate fast, but test SGD with momentum before final training
Your model trained with Adam is close to final and the remaining question is test accuracy, not speed of iteration.
mediumempirical finding - With Adam, use decoupled weight decay (AdamW), not the weight_decay L2 term
You add regularization to an Adam-trained model by setting weightdecay in torch.optim.Adam.
mediumdocumented fact - Exclude biases and normalization scales from weight decay
You apply AdamW to a model with LayerNorm or BatchNorm by passing model.parameters() with one weight-decay value.
hardpractitioner heuristic - When a Transformer diverges early, check normalization placement and warmup
A Transformer trained from scratch shows loss spikes or NaNs in the first few hundred steps at a learning rate that works later in training.
hardempirical finding - Replace BatchNorm when memory forces very small batches
High-resolution inspection images or long sequences force a batch size of 2 to 8 per GPU in a CNN that uses BatchNorm.
mediumempirical finding - Make sure BatchNorm uses running statistics when you evaluate and serve
Validation accuracy differs between two scoring scripts, or a model scores the same image differently in a batch of one and in a full batch.
easydocumented fact - Turn dropout off at inference unless you are deliberately sampling
A deployed demand model returns slightly different predictions for the same request, and nobody changed the code.
easydocumented fact - Treat Monte Carlo dropout uncertainty as a signal to validate, not a calibrated interval
Stakeholders want an uncertainty band around a neural demand forecast and you consider keeping dropout on at inference and averaging many passes.
hardpractitioner heuristic - Early-stop neural training on the decision metric and restore the best checkpoint
A churn network uses early stopping on validation loss with default patience, and the final weights are whatever the last epoch produced.
mediumpractitioner heuristic - Overfit a tiny batch before launching full training
A new model or data pipeline is ready and you are about to start a multi-hour training run.
easypractitioner heuristic - Gradient-check custom backward code in double precision
You implement a custom autograd function or a hand-written backward pass, for example for a custom pricing-aware loss.
mediumdocumented fact - Log the global gradient norm and clip it when spikes precede loss blow-ups
Training loss on a recurrent or deep model occasionally jumps by orders of magnitude, sometimes ending in NaN.
mediumdocumented fact - Diagnose vanishing gradients with per-layer gradient norms, then fix the architecture
A deep plain network trains no better than a shallower one, or its early layers barely change from initialization.
mediumdocumented fact - Triage a NaN loss in a fixed order, starting with the data
The loss of a fraud model becomes NaN in epoch 3, and the team starts changing the architecture.
mediumpractitioner heuristic - Use a gradient scaler with float16 mixed precision and compare against a float32 run
You enable float16 autocast to cut memory and speed up training of an image or text model.
mediumdocumented fact - Unscale gradients before clipping when training with a gradient scaler
You add gradient clipping to a training loop that already uses GradScaler for mixed precision.
mediumdocumented fact - Seed every library inside each data-loader worker
Augmentations draw randomness from NumPy, Python random or another library inside a Dataset, and the DataLoader runs several workers.
mediumdocumented fact - Split by source item before augmenting, never after
You multiply a small labelled image set, such as photos of damaged baggage or defect images, with crops and flips, then split the enlarged set randomly.
mediumpractitioner heuristic - Compute input normalization statistics on the training split only
You standardize tabular inputs for a neural net with a mean and standard deviation computed on the full dataset before splitting.
easydocumented fact - Beat a tuned gradient-boosting baseline before shipping a neural net on tabular data
A team proposes a deep model for a tabular problem, such as predicting final bookings per flight from booking-curve, fare and calendar features.
easyempirical finding - Give the baseline and the neural model the same tuning budget
A comparison report shows a neural model beating logistic regression and boosting on a churn dataset.
mediumpractitioner heuristic - Report results over several seeds before claiming a neural improvement
A fine-tuned text classifier scores 1.2 points higher than the current one on a single run.
mediumempirical finding - Switch on deterministic GPU mode while bisecting a training regression
After a refactor, validation accuracy of a credit-risk network drops by half a point, and you want to find the commit responsible.
mediumdocumented fact - Compute CNN feature-map sizes on paper before writing the classifier head
You stack convolution and pooling layers for an image task and hard-code the input size of the first linear layer.
easydocumented fact - Watch the flatten-to-dense layer, not the convolutions, for parameter blow-up
A small CNN for shelf images or X-ray triage has far more parameters than expected and overfits quickly.
mediumdocumented fact - Reproduce the pretrained model's preprocessing exactly before fine-tuning
You load a pretrained image or text backbone and feed it data prepared by your own pipeline.
easypractitioner heuristic - Start transfer learning with a linear probe, then fine-tune if it pays
You have a few thousand labelled examples and a pretrained backbone, and must decide whether to freeze it or fine-tune it.
mediumempirical finding - Unfreeze gradually and use smaller learning rates for lower layers on small text datasets
Fine-tuning a pretrained language model on a few hundred labelled complaint emails gives unstable results that forget general language knowledge.
mediumempirical finding - When BERT-style fine-tuning fails on some seeds, train longer with a smaller learning rate
Fine-tuning an encoder on a small classification set succeeds on some seeds and collapses to majority-class predictions on others.
hardempirical finding - Keep a retention test set to measure what fine-tuning made the model forget
You fine-tune a pretrained defect-detection or document model on a new product line and it improves on that line.
mediumempirical finding - Do not expect a plain RNN to carry information across hundreds of steps
You model long sequences, such as months of daily meter readings or long support transcripts, with a vanilla RNN.
mediumdocumented fact - Compare a sequence model of booking curves against lag features in a GBM
You want an LSTM or Transformer to forecast final bookings from the sequence of booking-curve snapshots per flight.
mediumpractitioner heuristic - Mask padding in attention, pooling and the loss, then test length invariance
You batch variable-length texts or event sequences by padding them to the longest item.
mediumdocumented fact - Use causal masking when the prediction must not see later tokens or events
You use a bidirectional encoder or an unmasked Transformer over a customer's event stream to predict their next action in real time.
mediumdocumented fact - Budget attention cost by sequence length before raising the context limit
Classifying long maintenance logs or contracts, someone proposes raising the maximum length from 512 to 4,096 tokens.
mediumdocumented fact - Do not present attention heatmaps as the reason for a prediction
A stakeholder asks why the model flagged a loan application or complaint, and the team shows an attention heatmap over the input words.
mediumempirical finding - Deduplicate near-identical texts across splits before trusting a text score
A complaint or FAQ dataset contains forwarded emails, copy-pasted templates and resubmitted tickets, and you split it randomly.
mediumempirical finding - Compare tokenizer fertility before choosing a multilingual encoder for a non-English task
You must classify Turkish customer complaints and are choosing between a multilingual encoder and a model pretrained mainly on Turkish text.
mediumempirical finding - Inspect how the tokenizer splits your domain codes and numbers
Inputs contain flight numbers, fare-class codes, booking references, part numbers or ICD codes, and a pretrained model handles them poorly.
mediumpractitioner heuristic - Prefer loss weighting to duplicating rare texts when classes are imbalanced
In a ticket classifier, safety-related complaints are 0.5% of messages and the model rarely predicts them.
mediumpractitioner heuristic - Expect large networks to memorize noisy training labels and plan for it
Training labels come from agents' quick tags and you train a large model for many epochs until training accuracy nears 100%.
mediumempirical finding - Score embeddings with the similarity function the model was trained for
You build semantic search over fare-rule documents and pick cosine similarity by habit for an embedding model you have not read about.
easydocumented fact - Watch for vocabulary drift in embedding-based models and adapt the encoder only when it pays
A complaint-topic clustering or classification model built on general-purpose sentence embeddings degrades after a new loyalty programme and new fare products introduce unfamiliar jargon.
hardempirical finding - Judge custom-trained embeddings by downstream lift, not by nearest-neighbour anecdotes
You trained product or route embeddings from co-purchase or co-search sequences and the team validates them by listing the nearest neighbours of a few favourite items.
mediumpractitioner heuristic - Re-embed the whole corpus whenever the embedding model or its preprocessing changes
You upgrade the embedding model for new documents but keep the existing vector index built with the old model.
mediumpractitioner heuristic - Run the first training step on the largest batch and longest sequence you will see
A text model trains with dynamic padding, and batch memory depends on the longest document in each batch.
easypractitioner heuristic - Trade compute for memory with activation checkpointing before shrinking the model
Training runs out of GPU memory as you increase sequence length or batch size, and the plan is to switch to a smaller model.
mediumdocumented fact - Measure GPU memory with the framework's allocator counters, not nvidia-smi alone
nvidia-smi shows a GPU almost full after training steps, and the team concludes the model cannot fit a larger batch.
easydocumented fact - Scale the loss and the schedule correctly when you accumulate gradients
You emulate a batch of 256 with 8 micro-batches of 32 because the full batch does not fit in memory.
hardpractitioner heuristic - Profile the input pipeline when GPU utilization stays low
A GPU training job runs slowly and GPU utilization hovers well below full while the CPU is busy.
easypractitioner heuristic - Use label smoothing knowing it changes calibration and distillation
You turn on label smoothing for a classifier that will later serve as the teacher for a smaller distilled model.
hardempirical finding - Route low-confidence predictions to people, but validate the confidence signal first
A claims-intake text classifier will auto-route confident predictions and send uncertain ones to human agents.
mediumempirical finding - Fix the total step budget before you choose a decaying learning-rate schedule
You pick a cosine or linear decay for a run and later decide to train "a few more epochs" because validation is still improving.
mediumpractitioner heuristic - Call the scheduler after the optimizer and decide whether it steps per batch or per epoch
A schedule that should warm up over 500 steps is configured through a scheduler object, and the training loop was written by combining examples from different library versions.
easydocumented fact - Checkpoint the optimizer, scheduler and step counter, not only the weights
A long training job is preempted and you resume it from a file that holds only the model weights.
easydocumented fact - Start residual blocks as the identity by zero-initialising their last scale
A deep residual network trains with a very large batch, or its loss is erratic during the first hundred steps even after warmup.
hardempirical finding - Keep a frozen backbone's BatchNorm layers in inference mode while you fine-tune
You freeze a pretrained image backbone, train a new head on a few thousand labelled images, then unfreeze the top blocks.
mediumdocumented fact - Measure steps to target at several batch sizes before paying for bigger batches
A team wants to move a training job from 4 to 32 accelerators and assumes the run will finish about 8 times sooner.
hardempirical finding - Estimate the gradient noise scale to see when a larger batch has stopped helping
Training is stable at your current batch size and you are deciding whether to double it, but a full sweep is too expensive.
hardempirical finding - Do not place dropout directly before a BatchNorm layer
A convolutional or dense network has a dropout layer immediately followed by BatchNorm and validation accuracy is slightly worse in evaluation mode than the training curve suggests.
mediumempirical finding - When you add mixup, treat the targets as soft and change the metrics accordingly
You add mixup to an image or tabular network that overfits a few thousand labelled rows, and keep the old accuracy and threshold code.
mediumempirical finding - Look at a labelled grid of augmented samples before you train on them
You add random crops, flips, jitter or noise to a training pipeline and plan to judge the result only by validation accuracy.
easypractitioner heuristic - Recalibrate after adopting focal loss for an imbalanced classifier
A rare-event network (for example, a synthetic no-show flag at 2 percent prevalence) is trained with focal loss and its scores are fed into a cost-based threshold.
mediumempirical finding - Put the task losses on one scale before adding them in a multi-task network
A shared network predicts a revenue amount in euros and a cancellation probability, and the total loss is the plain sum of a squared error and a cross-entropy.
mediumempirical finding - Rank training examples by out-of-fold loss to find labels worth re-checking
A classifier plateaus below what the data seems to allow, and the labels come from a rule, a vendor or a stressed annotation team.
mediumempirical finding - Do not quote one expected-calibration-error number without its binning and a reliability diagram
A report compares two networks by a single ECE value computed with ten equal-width bins on a test set of 800 rows.
mediumempirical finding - Prefer a small deep ensemble over MC dropout when uncertainty must hold up
A neural classifier's confidence will gate automated decisions, and you can afford several times the training and inference compute.
mediumempirical finding - Re-check calibration on shifted data, because a temperature fitted in distribution may not travel
A temperature-scaled network is deployed to a new region, season or device mix and its probabilities feed a threshold chosen on the old validation split.
hardempirical finding - Average the weights of late checkpoints, then recompute BatchNorm statistics
Validation loss of a small network fluctuates by a few percent between neighbouring epochs and you must pick one checkpoint to ship.
mediumempirical finding - Interpolate fine-tuned and original weights to keep robustness a fine-tune would erase
Fine-tuning a strong pretrained model lifts accuracy on your validation split, but the model is used on inputs that look different from the fine-tuning data.
hardempirical finding - Vary the initialisation seed and the data-order seed separately to see which one moves the metric
Two runs of the same configuration differ by half a point and you do not know whether to blame initialisation, shuffling, augmentation or hardware.
mediumempirical finding - Write a run manifest that lets a stranger repeat the experiment
A model that scored well three weeks ago cannot be rebuilt: the notebook, the library versions and the data extract have all changed.
easypractitioner heuristic - Register submodules in ModuleList or ModuleDict, never in a plain Python list
A model builds a variable number of layers in a loop and stores them in a Python list or dict attribute.
easydocumented fact - Watch the update-to-weight ratio per layer to size a learning rate
Training is slow or erratic and you cannot tell whether the learning rate is too small for some layers or too large for others.
mediumpractitioner heuristic - Shuffle the training labels once and confirm the validation score collapses to chance
A network scores surprisingly high on a new dataset, and the pipeline joins several tables and applies preprocessing before the split.
mediumpractitioner heuristic - Train once with the inputs zeroed to prove the model uses its features
A network reaches a decent score, but its loss curve looks similar to what a constant prediction would give.
easypractitioner heuristic - Choose bfloat16 or float16 by range, not by speed alone, and remember float16 needs a scaler
You enable mixed precision on a model that was pretrained or developed in bfloat16, or you move a working float32 job to float16 for speed.
mediumdocumented fact - Keep a float32 master copy of the weights, and compute losses and reductions in float32
You cast an entire model and optimizer to half precision to fit a bigger batch, and the loss stops improving after the learning rate decays.
mediumdocumented fact - Include Adam's epsilon in the search space before ranking optimizers
A comparison of SGD with momentum and Adam is run with default settings, and Adam loses by a point on a tabular-style network.
hardempirical finding - Get a model that overfits the full training set before you add regularisation
A new network underperforms and you immediately add dropout, weight decay and augmentation to it all at once.
easypractitioner heuristic - Shuffle the training loader explicitly, because the default does not
A DataLoader is built from rows exported sorted by route, date or label, and training loss is jagged with a repeating pattern of the same length as the epoch.
easydocumented fact - Embed numerical features before the MLP when a tabular network trails gradient boosting
A plain MLP on raw standardised numerical features of a tabular problem (fare, days to departure, load factor) underperforms a gradient-boosted model by a clear margin.
hardempirical finding - Fit a learning curve over training-set size before buying more labels or a larger model
A team plans to triple labelling spend because validation error is still above target, without knowing how error has moved with data so far.
mediumempirical finding - Do not stop at the first rise in validation loss without checking a longer horizon
An over-parameterised network's validation error rises in mid-training, then, with patience set to five epochs, training halts and a smaller model is chosen instead.
hardempirical finding - Prune weak configurations early with successive halving when each run is costly
A search over learning rate, width and regularisation needs 100 candidate configurations, and each run takes a day on a shared GPU.
mediumempirical finding - Synchronise BatchNorm statistics when the per-device batch becomes small
You scale a job from one GPU with batch 64 to eight GPUs with 8 examples each, using data-parallel training, and keep BatchNorm layers.
mediumdocumented fact - Normalise Unicode text to one form before tokenising, hashing or deduplicating
Turkish messages arrive from several apps and keyboards, and the same letter with a cedilla or breve is sometimes one precomposed character and sometimes a base letter plus a combining mark.
easydocumented fact - Use feature hashing when the vocabulary keeps growing, and accept what it gives up
A ticket classifier is refit nightly and a fitted vocabulary keeps meeting new product names, flight codes and typos, while the vocabulary dictionary grows in memory.
mediumdocumented fact - Choose which segment of a text pair to truncate instead of accepting the default strategy
A BERT-style pair classifier takes a customer message and a long policy clause with maxlength=128 and truncation=True.
mediumdocumented fact - Count how often each item shows up in top-k lists to catch hub documents in embedding search
In a vector search over support articles, the same handful of articles appears in the top 10 for many unrelated queries.
hardpractitioner heuristic - Split text data by thread or customer so quoted replies and personal style cannot cross the boundary
Support emails are split by message at random, though each thread has several messages that quote earlier ones and come from the same customer.
mediumempirical finding - Exclude text written after the outcome, such as resolution notes and closing remarks
You predict at ticket creation whether a complaint will be escalated, using a text column that holds the full conversation including later agent notes.
mediumdocumented fact - Reweight to the production class prior when labelling effort was split evenly across intents
A labelling budget was split equally over six intents so each has the same count, while production traffic has one dominant intent and several rare ones.
harddocumented fact - Keep invariance and directional tests for a text classifier next to its accuracy report
A complaint-priority classifier passes its test set and then fails on trivial rewrites in the first demo.
mediumpractitioner heuristic - Check mask polarity at every library boundary, because True can mean attend or ignore
Custom transformer code combines a Hugging Face attentionmask with nn.MultiheadAttention or scaleddotproductattention.
mediumdocumented fact - Keep the 1/sqrt(d_k) factor when you write attention by hand, and log how peaked the weights are
You hand-write scaled dot-product attention for a small sequence model and drop the division as cosmetic.
mediumdocumented fact - Shuffle token or event order at test time to prove a Transformer actually uses position
You build a Transformer over ordered booking events or sentences from scratch, and its validation score is decent.
mediumdocumented fact - Test on sequences longer than any seen in training before assuming the position scheme extends
A model trained on sessions of up to 128 events will see sessions of 400 events in production.
hardempirical finding - Guard against rows where every key is masked, because softmax over all minus-infinity is undefined
A batch contains an empty text or a fully padded sequence, and the attention scores of that row are all filled with minus infinity before the softmax.
mediumpractitioner heuristic - Left-pad batched prompts for decoder-only generation and confirm batched outputs match single runs
You batch prompts of different lengths for a decoder-only model, using the tokenizer's default right padding.
easydocumented fact - Set padding_idx on embeddings and still mask the pool, or padding length leaks into the features
A text model uses nn.Embedding with pad id 0 and mean-pools token vectors over the padded length.
easydocumented fact - Read the last real time step of a padded RNN batch, not position minus one
An LSTM classifier over right-padded batches uses output[:, -1] as the sequence summary.
mediumpractitioner heuristic - Detach the hidden state between truncated-BPTT chunks and treat chunk length as the learnable dependency horizon
You train an RNN on one long stream by carrying the hidden state across consecutive chunks.
harddocumented fact - Use a 1-D CNN with max-pooling as a fast pattern-detector baseline for long sequences
A label depends on short local patterns in long event or text sequences, and recurrent models are slow to train.
mediumpractitioner heuristic - Do not assume a CNN uses its full theoretical receptive field
You size a CNN on paper so its theoretical receptive field covers the whole image or sensor window and conclude the context is covered.
hardempirical finding - Probe a CNN with texture and capture shifts before moving it to a new camera or catalogue
A product-image or defect CNN trained under one camera and lighting setup is moved to another source, such as customer photos instead of catalogue shots.
mediumempirical finding