Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Feed raw logits to CrossEntropyLoss, never softmax outputs
A multi-class model ends in nn.Softmax and its output goes straight into nn.CrossEntropyLoss.
easydocumented fact - Pick softmax or independent sigmoids by asking whether labels can co-occur
You classify flight disruption reports into causes (weather, crew, technical, ATC) and some reports cite several causes.
mediumdocumented fact - With Adam, use decoupled weight decay (AdamW), not the weight_decay L2 term
You add regularization to an Adam-trained model by setting weightdecay in torch.optim.Adam.
mediumdocumented fact - Exclude biases and normalization scales from weight decay
You apply AdamW to a model with LayerNorm or BatchNorm by passing model.parameters() with one weight-decay value.
hardpractitioner heuristic - When a Transformer diverges early, check normalization placement and warmup
A Transformer trained from scratch shows loss spikes or NaNs in the first few hundred steps at a learning rate that works later in training.
hardempirical finding - Overfit a tiny batch before launching full training
A new model or data pipeline is ready and you are about to start a multi-hour training run.
easypractitioner heuristic - Report results over several seeds before claiming a neural improvement
A fine-tuned text classifier scores 1.2 points higher than the current one on a single run.
mediumempirical finding - Reproduce the pretrained model's preprocessing exactly before fine-tuning
You load a pretrained image or text backbone and feed it data prepared by your own pipeline.
easypractitioner heuristic - Start transfer learning with a linear probe, then fine-tune if it pays
You have a few thousand labelled examples and a pretrained backbone, and must decide whether to freeze it or fine-tune it.
mediumempirical finding - Unfreeze gradually and use smaller learning rates for lower layers on small text datasets
Fine-tuning a pretrained language model on a few hundred labelled complaint emails gives unstable results that forget general language knowledge.
mediumempirical finding - When BERT-style fine-tuning fails on some seeds, train longer with a smaller learning rate
Fine-tuning an encoder on a small classification set succeeds on some seeds and collapses to majority-class predictions on others.
hardempirical finding - Keep a retention test set to measure what fine-tuning made the model forget
You fine-tune a pretrained defect-detection or document model on a new product line and it improves on that line.
mediumempirical finding - Mask padding in attention, pooling and the loss, then test length invariance
You batch variable-length texts or event sequences by padding them to the longest item.
mediumdocumented fact - Use causal masking when the prediction must not see later tokens or events
You use a bidirectional encoder or an unmasked Transformer over a customer's event stream to predict their next action in real time.
mediumdocumented fact - Budget attention cost by sequence length before raising the context limit
Classifying long maintenance logs or contracts, someone proposes raising the maximum length from 512 to 4,096 tokens.
mediumdocumented fact - Do not present attention heatmaps as the reason for a prediction
A stakeholder asks why the model flagged a loan application or complaint, and the team shows an attention heatmap over the input words.
mediumempirical finding - Start text classification with TF-IDF and a linear model
You need to route airline customer complaints into 8 queues and the first idea is to fine-tune a Transformer.
easyempirical finding - Deduplicate near-identical texts across splits before trusting a text score
A complaint or FAQ dataset contains forwarded emails, copy-pasted templates and resubmitted tickets, and you split it randomly.
mediumempirical finding - Check whether a text model learned the template instead of the content
A ticket classifier scores very high, and many tickets come from web forms, partner systems or agents with their own boilerplate.
mediumpractitioner heuristic - Compare tokenizer fertility before choosing a multilingual encoder for a non-English task
You must classify Turkish customer complaints and are choosing between a multilingual encoder and a model pretrained mainly on Turkish text.
mediumempirical finding - Inspect how the tokenizer splits your domain codes and numbers
Inputs contain flight numbers, fare-class codes, booking references, part numbers or ICD codes, and a pretrained model handles them poorly.
mediumpractitioner heuristic - Check where the label evidence sits before truncating long texts
Insurance claim narratives or incident reports often exceed the encoder's maximum length and are cut at the limit.
mediumpractitioner heuristic - Report per-class recall and macro averages for long-tail intent classifiers
A banking chatbot intent model reports 94% accuracy over 60 intents, and most traffic falls into five intents.
easypractitioner heuristic - Audit test-set labels before ranking models that differ by a point or two
Two text classifiers differ by 1.5 points on a test set labelled quickly by several annotators.
mediumempirical finding - Score embeddings with the similarity function the model was trained for
You build semantic search over fare-rule documents and pick cosine similarity by habit for an embedding model you have not read about.
easydocumented fact - Do not reuse a cosine-similarity threshold across embedding models
A duplicate-ticket detector flags pairs above cosine 0.8, and the team swaps in a new embedding model without changing the threshold.
mediumpractitioner heuristic - Watch for vocabulary drift in embedding-based models and adapt the encoder only when it pays
A complaint-topic clustering or classification model built on general-purpose sentence embeddings degrades after a new loyalty programme and new fare products introduce unfamiliar jargon.
hardempirical finding - Do not treat a t-SNE or UMAP picture of embeddings as evidence of separation
A slide shows customer-message embeddings in 2D with neat colored clusters and claims the model separates segments.
easydocumented fact - Re-embed the whole corpus whenever the embedding model or its preprocessing changes
You upgrade the embedding model for new documents but keep the existing vector index built with the old model.
mediumpractitioner heuristic - Use contextual embeddings when word sense changes the label
You classify short airline messages with averaged static word vectors, and words like "hold" (baggage hold, on hold, hold a seat) carry the meaning.
mediumempirical finding - Run the first training step on the largest batch and longest sequence you will see
A text model trains with dynamic padding, and batch memory depends on the longest document in each batch.
easypractitioner heuristic - Use label smoothing knowing it changes calibration and distillation
You turn on label smoothing for a classifier that will later serve as the teacher for a smaller distilled model.
hardempirical finding - Route low-confidence predictions to people, but validate the confidence signal first
A claims-intake text classifier will auto-route confident predictions and send uncertain ones to human agents.
mediumempirical finding - Keep a frozen backbone's BatchNorm layers in inference mode while you fine-tune
You freeze a pretrained image backbone, train a new head on a few thousand labelled images, then unfreeze the top blocks.
mediumdocumented fact - Interpolate fine-tuned and original weights to keep robustness a fine-tune would erase
Fine-tuning a strong pretrained model lifts accuracy on your validation split, but the model is used on inputs that look different from the fine-tuning data.
hardempirical finding - Test character n-grams before building a stemmer for agglutinative Turkish text
A Turkish complaint classifier uses word-level TF-IDF, and one stem shows up in many surface forms because suffixes stack onto it.
mediumpractitioner heuristic - Measure how much Turkish text is typed without diacritics before trusting a clean benchmark score
A model trained on correctly accented Turkish complaints will read chat and mobile messages where people type c, s, g, i, o, u for the letters with diacritics.
mediumpractitioner heuristic - Choose which segment of a text pair to truncate instead of accepting the default strategy
A BERT-style pair classifier takes a customer message and a long policy clause with maxlength=128 and truncation=True.
mediumdocumented fact - Centre or whiten sentence embeddings before nearest-neighbour search and compare with the raw vectors
Scores between unrelated texts from a BERT-based encoder are all high, and every query seems to retrieve similar neighbours.
hardpractitioner heuristic - Count how often each item shows up in top-k lists to catch hub documents in embedding search
In a vector search over support articles, the same handful of articles appears in the top 10 for many unrelated queries.
hardpractitioner heuristic - Measure pretrained word-vector coverage on your own tokens, by slice, before relying on static embeddings
GloVe or word2vec vectors feed a complaint classifier, and the text is full of flight codes, brand names and inflected Turkish forms.
easypractitioner heuristic - Build a contrast set before letting embedding similarity merge texts automatically
A duplicate-ticket merger links two tickets whenever the cosine similarity of their embeddings passes a threshold chosen on random pairs.
mediumpractitioner heuristic - Do not use raw BERT [CLS] or mean-pooled outputs as sentence embeddings for similarity
A prototype embeds sentences with a base BERT checkpoint, takes the [CLS] vector or the mean of the last layer, and ranks by cosine.
easyempirical finding - Evaluate zero-shot cross-lingual transfer on labelled Turkish data before launching in Turkish
A multilingual encoder is fine-tuned on English tickets and expected to work on Turkish tickets with no Turkish labels.
mediumempirical finding - Report metrics per detected language when messages mix Turkish and English
Customer messages for an airline are Turkish, English and mixed, and one classifier reports a single accuracy.
mediumpractitioner heuristic - Compare frozen features and fine-tuning at several training sizes before committing
A team has 600 labelled Turkish complaints and argues about frozen embeddings with a linear head versus fine-tuning.
mediumempirical finding - Count selection data as part of the budget when you only own a few dozen labels
A few-shot text classifier is reported at a score tuned on prompts and learning rates using a large extra validation set, though production will have only 60 labels.
hardempirical finding - Keep per-item annotator votes and test soft labels before collapsing them to a majority
Three annotators label each tone or intent example, the pipeline keeps the majority label, and a third of the items were split 2 to 1.
hardempirical finding - Hold out annotators, not only items, when a few people produced most of the labels or texts
A tone classifier is trained on labels from five agents, two of whom labelled 70 percent of the rows, and the test split is random.
mediumempirical finding - Run a partial-input baseline to expose annotation artifacts before celebrating a text score
A text-pair or template-labelled dataset scores high and you suspect the label writers left stylistic traces.
mediumempirical finding - Keep invariance and directional tests for a text classifier next to its accuracy report
A complaint-priority classifier passes its test set and then fails on trivial rewrites in the first demo.
mediumpractitioner heuristic - Check mask polarity at every library boundary, because True can mean attend or ignore
Custom transformer code combines a Hugging Face attentionmask with nn.MultiheadAttention or scaleddotproductattention.
mediumdocumented fact - Keep the 1/sqrt(d_k) factor when you write attention by hand, and log how peaked the weights are
You hand-write scaled dot-product attention for a small sequence model and drop the division as cosmetic.
mediumdocumented fact - Shuffle token or event order at test time to prove a Transformer actually uses position
You build a Transformer over ordered booking events or sentences from scratch, and its validation score is decent.
mediumdocumented fact - Test on sequences longer than any seen in training before assuming the position scheme extends
A model trained on sessions of up to 128 events will see sessions of 400 events in production.
hardempirical finding - Left-pad batched prompts for decoder-only generation and confirm batched outputs match single runs
You batch prompts of different lengths for a decoder-only model, using the tokenizer's default right padding.
easydocumented fact - Exclude duplicate positives from a batch when training a retriever with in-batch negatives
A bi-encoder is fine-tuned on question and answer pairs with in-batch negatives, and several questions share the same answer or the same question appears twice.
mediumpractitioner heuristic - Realign word-level labels to subword tokens before token classification and score only the labelled positions
A tagger for flight numbers or airport names is trained on word-level labels, but the tokenizer splits rare or inflected words into pieces and adds special tokens.
mediumdocumented fact