Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Pair every target metric with a counter-metric that exposes gaming
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
easypractitioner heuristic - Write chart titles that state the finding and its scope, not the variables plotted
A slide is titled "Bookings by month, 2023-2024" and the presenter explains the point verbally.
easypractitioner heuristic - Treat public-benchmark scores of forecasting foundation models as possibly contaminated
A time-series foundation model is proposed because it tops a public leaderboard.
mediumdocumented fact - Canary a new model on a small traffic slice with pre-agreed rollback criteria
A retrained model is ready, and the plan is to switch all traffic at once on Monday.
mediumdocumented fact - Keep rollback one alias change away and test it before you need it
A new model version misbehaves, and rollback requires rebuilding the old container and feature pipeline.
easydocumented fact - Gate releases on a frozen golden set with tolerances sized to run-to-run noise
A model-quality CI check fails randomly about one run in five, and engineers start rerunning until it passes.
mediumpractitioner heuristic - Instrument a model service with the golden signals plus model-specific signals
A model endpoint has a health check that returns OK, and users report slow and odd recommendations.
easydocumented fact - Plan how deletion requests propagate to snapshots, features and retraining
A customer's data is deleted from the operational database, but it remains in last quarter's training snapshot and feature tables.
hardpractitioner heuristic - Check model and dataset licences before a prototype becomes a product
A prototype uses an open-weights model and a public dataset, and the business wants to launch it commercially.
easydocumented fact - Ship a model card with intended use and performance by relevant slice
A claims-triage model is handed to an operations team with only an overall accuracy figure.
easypractitioner heuristic - Scale governance effort to the risk tier of each model
Every model, from an internal dashboard forecast to a credit decision engine, goes through the same three-month review.
mediumpractitioner heuristic - Price inference per thousand predictions before choosing GPU serving
A team proposes GPU instances to serve a gradient-boosting model that handles 50 requests per second.
mediumpractitioner heuristic - Compare managed and self-hosted ML platforms on total cost including people
Self-hosting an open-source serving stack looks cheaper than a managed service on the cloud bill.
mediumpractitioner heuristic - Track unit cost per decision, not just the monthly cloud bill
Monthly ML infrastructure cost rose 40%, and nobody can say whether it is growth, waste or a new model.
easypractitioner heuristic - Negotiate an ambiguous AI request down to one decision, one metric and a baseline
A director asks the team to 'use AI to optimise the network' within a quarter.
easypractitioner heuristic - Refresh internal benchmark sets so teams stop tuning to a fixed test
Every quarter, teams report improvements on the same internal evaluation set used for two years.
mediumdocumented fact - Estimate the value of a model with a back-of-envelope calculation before building it
A team proposes a three-month project to improve a no-show prediction model.
easypractitioner heuristic - Keep a decision and assumption log so a model survives handover
A modelling project is handed to a new team, and the reasons for its target definition, exclusions and thresholds exist only in old chat threads.
easypractitioner heuristic - Feed raw logits to CrossEntropyLoss, never softmax outputs
A multi-class model ends in nn.Softmax and its output goes straight into nn.CrossEntropyLoss.
easydocumented fact - Pick softmax or independent sigmoids by asking whether labels can co-occur
You classify flight disruption reports into causes (weather, crew, technical, ATC) and some reports cite several causes.
mediumdocumented fact - With Adam, use decoupled weight decay (AdamW), not the weight_decay L2 term
You add regularization to an Adam-trained model by setting weightdecay in torch.optim.Adam.
mediumdocumented fact - Exclude biases and normalization scales from weight decay
You apply AdamW to a model with LayerNorm or BatchNorm by passing model.parameters() with one weight-decay value.
hardpractitioner heuristic - When a Transformer diverges early, check normalization placement and warmup
A Transformer trained from scratch shows loss spikes or NaNs in the first few hundred steps at a learning rate that works later in training.
hardempirical finding - Overfit a tiny batch before launching full training
A new model or data pipeline is ready and you are about to start a multi-hour training run.
easypractitioner heuristic - Report results over several seeds before claiming a neural improvement
A fine-tuned text classifier scores 1.2 points higher than the current one on a single run.
mediumempirical finding - Reproduce the pretrained model's preprocessing exactly before fine-tuning
You load a pretrained image or text backbone and feed it data prepared by your own pipeline.
easypractitioner heuristic - Start transfer learning with a linear probe, then fine-tune if it pays
You have a few thousand labelled examples and a pretrained backbone, and must decide whether to freeze it or fine-tune it.
mediumempirical finding - Unfreeze gradually and use smaller learning rates for lower layers on small text datasets
Fine-tuning a pretrained language model on a few hundred labelled complaint emails gives unstable results that forget general language knowledge.
mediumempirical finding - When BERT-style fine-tuning fails on some seeds, train longer with a smaller learning rate
Fine-tuning an encoder on a small classification set succeeds on some seeds and collapses to majority-class predictions on others.
hardempirical finding - Keep a retention test set to measure what fine-tuning made the model forget
You fine-tune a pretrained defect-detection or document model on a new product line and it improves on that line.
mediumempirical finding - Mask padding in attention, pooling and the loss, then test length invariance
You batch variable-length texts or event sequences by padding them to the longest item.
mediumdocumented fact - Use causal masking when the prediction must not see later tokens or events
You use a bidirectional encoder or an unmasked Transformer over a customer's event stream to predict their next action in real time.
mediumdocumented fact - Budget attention cost by sequence length before raising the context limit
Classifying long maintenance logs or contracts, someone proposes raising the maximum length from 512 to 4,096 tokens.
mediumdocumented fact - Do not present attention heatmaps as the reason for a prediction
A stakeholder asks why the model flagged a loan application or complaint, and the team shows an attention heatmap over the input words.
mediumempirical finding - Start text classification with TF-IDF and a linear model
You need to route airline customer complaints into 8 queues and the first idea is to fine-tune a Transformer.
easyempirical finding - Deduplicate near-identical texts across splits before trusting a text score
A complaint or FAQ dataset contains forwarded emails, copy-pasted templates and resubmitted tickets, and you split it randomly.
mediumempirical finding - Check whether a text model learned the template instead of the content
A ticket classifier scores very high, and many tickets come from web forms, partner systems or agents with their own boilerplate.
mediumpractitioner heuristic - Compare tokenizer fertility before choosing a multilingual encoder for a non-English task
You must classify Turkish customer complaints and are choosing between a multilingual encoder and a model pretrained mainly on Turkish text.
mediumempirical finding - Inspect how the tokenizer splits your domain codes and numbers
Inputs contain flight numbers, fare-class codes, booking references, part numbers or ICD codes, and a pretrained model handles them poorly.
mediumpractitioner heuristic - Check where the label evidence sits before truncating long texts
Insurance claim narratives or incident reports often exceed the encoder's maximum length and are cut at the limit.
mediumpractitioner heuristic - Report per-class recall and macro averages for long-tail intent classifiers
A banking chatbot intent model reports 94% accuracy over 60 intents, and most traffic falls into five intents.
easypractitioner heuristic - Audit test-set labels before ranking models that differ by a point or two
Two text classifiers differ by 1.5 points on a test set labelled quickly by several annotators.
mediumempirical finding - Score embeddings with the similarity function the model was trained for
You build semantic search over fare-rule documents and pick cosine similarity by habit for an embedding model you have not read about.
easydocumented fact - Do not reuse a cosine-similarity threshold across embedding models
A duplicate-ticket detector flags pairs above cosine 0.8, and the team swaps in a new embedding model without changing the threshold.
mediumpractitioner heuristic - Watch for vocabulary drift in embedding-based models and adapt the encoder only when it pays
A complaint-topic clustering or classification model built on general-purpose sentence embeddings degrades after a new loyalty programme and new fare products introduce unfamiliar jargon.
hardempirical finding - Do not treat a t-SNE or UMAP picture of embeddings as evidence of separation
A slide shows customer-message embeddings in 2D with neat colored clusters and claims the model separates segments.
easydocumented fact - Re-embed the whole corpus whenever the embedding model or its preprocessing changes
You upgrade the embedding model for new documents but keep the existing vector index built with the old model.
mediumpractitioner heuristic - Use contextual embeddings when word sense changes the label
You classify short airline messages with averaged static word vectors, and words like "hold" (baggage hold, on hold, hold a seat) carry the meaning.
mediumempirical finding - Run the first training step on the largest batch and longest sequence you will see
A text model trains with dynamic padding, and batch memory depends on the longest document in each batch.
easypractitioner heuristic - Use label smoothing knowing it changes calibration and distillation
You turn on label smoothing for a classifier that will later serve as the teacher for a smaller distilled model.
hardempirical finding - Route low-confidence predictions to people, but validate the confidence signal first
A claims-intake text classifier will auto-route confident predictions and send uncertain ones to human agents.
mediumempirical finding - Count tokens per language before sizing prompts, budgets and context limits
An airline assistant must answer customer messages in English, Turkish, Arabic and German, and the team sized its context and cost budget from English test prompts.
easyempirical finding - Test effective context at your own input lengths instead of trusting the advertised window
A team plans to paste a whole ground-handling contract (a hypothetical 300 pages) into one prompt because it fits inside the documented context window.
mediumempirical finding - Supply volatile facts at request time because parametric knowledge is frozen at training
Agents ask a chat model about this season's baggage allowance or a fare rule that changed last month.
easydocumented fact - Slice LLM accuracy by entity popularity because long-tail facts fail first
A travel assistant answers well about major hubs in demos, and the team plans to launch it for all airports in the network.
mediumempirical finding - Separate total from active parameters when sizing mixture-of-experts models
A team compares a mixture-of-experts (MoE) model with a dense model for self-hosting using only the headline parameter count.
mediumdocumented fact - For fixed-label text classification, benchmark a small encoder before a generative LLM
A team wants a large generative model to tag two million complaint emails a month into 12 fixed categories.
mediumpractitioner heuristic - Match checkpoint type and chat template to the job before judging an open model
An engineer downloads open model weights, sends plain instructions and gets rambling, off-task text.
easydocumented fact - Set decoding parameters per task instead of one global temperature
One shared configuration sets a high temperature for every LLM call, from marketing copy to fare-rule extraction.
easyempirical finding - Repeat evaluation runs because temperature 0 does not guarantee identical outputs
A prompt change is approved because a single evaluation run at temperature 0 scored two points higher than the old prompt.
easydocumented fact - Use majority voting over sampled answers for verifiable reasoning tasks, and price it
An LLM computes refund amounts from multi-step fare rules and is wrong on a noticeable share of eval cases with a single chain-of-thought answer.
mediumempirical finding - Check the stop reason before parsing, and treat truncated output as a failure
A batch job extracts structured itinerary changes and occasionally receives JSON that ends mid-object.
easydocumented fact - Triage hallucinations as contradictions or unsupported additions because the fixes differ
Reviewers label 40 generated summaries of delay-compensation cases simply as hallucinated or fine.
mediumpractitioner heuristic - Score long answers claim by claim instead of with one overall verdict
A disruption-report generator writes 300-word summaries and reviewers mark each summary acceptable or not.
mediumempirical finding - Use disagreement across samples as a hallucination flag when no reference exists
An assistant drafts airport transit guidance for which no structured source of truth exists yet.
mediumempirical finding - Replace "check your answer" prompts with an external verifier
To cut errors, a team adds a second prompt asking the model to review and fix its own fare calculation.
mediumempirical finding - Give the model a scored way to abstain, or your eval rewards confident guessing
An eval for a policy assistant scores answers only as right or wrong, so a reply of insufficient information counts as wrong.
mediumpractitioner heuristic - Verify every identifier the model emits against the system of record
An assistant drafts rebooking notes that include flight numbers, fare basis codes and policy section numbers.
easypractitioner heuristic - Seed eval sets with questions that invite common misconceptions
A revenue-management help bot is evaluated only on questions written by the people who built it.
mediumempirical finding - Keep your hypothesis out of the prompt when asking an LLM to judge evidence
An analyst asks a model to confirm that a revenue drop on one route was caused by a competitor's new capacity.
mediumempirical finding - Do not treat a model's stated reasoning as an audit trail
A reviewer accepts an LLM's credit-limit recommendation because its step-by-step explanation looks sound.
hardempirical finding - Write success criteria and test cases before the first prompt revision
A team has rewritten a booking-change prompt eleven times, each time reacting to whichever example last went wrong.
easypractitioner heuristic - Balance and shuffle few-shot examples, then test several orders
A few-shot prompt for classifying refund requests lists five approve examples followed by one reject example.
mediumempirical finding - Use few-shot examples to show format and label space, not to teach new rules
A team tries to teach the model a new internal overbooking-compensation rule by adding three worked examples to the prompt.
mediumempirical finding - Evaluate prompts across several formatting variants before comparing models
Two candidate models are compared with one prompt template that was tuned on model A.
mediumempirical finding - Use provider-enforced schemas instead of asking for JSON in the prompt
A pipeline extracts passenger name, flight and requested change from emails by asking the model to reply in JSON, then parses with a try/except.
easydocumented fact - Validate business rules after the schema passes, because schemas only check shape
A structured-output extractor for refund requests returns perfectly valid objects, and the team stops checking them.
easypractitioner heuristic - Give every extracted field an explicit unknown value so the model is not forced to invent
An extraction schema requires a booking reference, a fare class and a travel date for every complaint email.
easypractitioner heuristic - Repair failed outputs with the validator's error message and a retry cap
A structured extraction fails business-rule validation on a few percent of documents, and an engineer adds code that silently coerces bad values into defaults.
easypractitioner heuristic - Prune long system prompts by ablating rules against the eval set
A customer-service system prompt has grown to dozens of rules added after individual complaints, some of them contradicting each other.
mediumpractitioner heuristic - Extract from long documents section by section with provenance, then aggregate in code
An analyst asks a model to list every penalty clause from a long supplier contract in one call.
mediumpractitioner heuristic - Separate instructions from pasted data with clear delimiters, but do not call it security
A prompt mixes instructions with a pasted customer email, and the model sometimes follows sentences inside the email as if they were instructions.
easypractitioner heuristic - Let the model reason before you force strict formats on reasoning-heavy tasks
A multi-step fare-difference calculation is moved into a strict JSON output mode, and accuracy on the eval set drops.
mediumempirical finding - Never draw few-shot demonstrations from the evaluation set
To improve a classification prompt quickly, an engineer copies the hardest misclassified eval cases into the prompt as examples.
mediumpractitioner heuristic - Supply context instead of personas, because role prompts do not add knowledge
A prompt starts by telling the model it is a world-class revenue-management expert, and the team expects better pricing analysis as a result.
easyempirical finding - Measure chain-of-thought prompting per task instead of switching it on everywhere
After reading that step-by-step prompting improves reasoning, a team adds it to every prompt, including simple classification.
easyempirical finding - Require a fine-tune to beat the best prompt on a frozen eval set before you own it
A team proposes fine-tuning a model for disruption-notice drafting after two weeks of prompt work.
mediumpractitioner heuristic - Deduplicate fine-tuning data against the eval set before training
Fine-tuning data is exported from the same ticket archive that supplied the eval set.
mediumpractitioner heuristic - Name the missing capability before choosing prompting, retrieval or fine-tuning
A stakeholder asks for the model to be fine-tuned on company data because answers about baggage policy are wrong.
mediumpractitioner heuristic - Do not fine-tune to inject facts that change; retrieve them instead
A team fine-tunes a model on this quarter's fare rules so agents can ask about them without retrieval.
mediumempirical finding - Fine-tune for consistent behaviour that long prompts cannot hold
A prompt for drafting delay notices needs a long style guide and many examples, yet tone and structure still drift across thousands of daily messages.
mediumpractitioner heuristic - Curate a small, consistent fine-tuning set before collecting a large noisy one
A team plans to fine-tune on 200,000 historical agent replies exported without review.
mediumempirical finding - Rerun safety and refusal tests after every fine-tune, even on benign data
A team fine-tunes an aligned model on harmless customer-service transcripts and evaluates only task quality.
mediumempirical finding - Consider continued pretraining only when you hold a large unlabelled domain corpus
An insurer's claims text is full of domain jargon, and a general model handles it poorly even after prompt work.
hardempirical finding - Anchor synthetic training data to real data and track its provenance
To save labelling cost, a team trains each new model version on outputs generated by the previous version.
mediumempirical finding - Do the LoRA parameter arithmetic before planning hardware
A team asks whether LoRA lets them fine-tune a model on the GPU they already have.
mediumdocumented fact - Expect LoRA to learn less and forget less than full fine-tuning
A team uses LoRA to teach a model a large new skill domain, such as a proprietary programming language, and gains are small.
hardempirical finding - Evaluate the exact artefact you will serve, not the training checkpoint
A LoRA adapter scores well during training, then is merged into the base weights and quantized for serving without re-evaluation.
easypractitioner heuristic - Judge a compressed model on your tasks, not on perplexity alone
A self-hosted model is quantized or pruned to cut serving memory, and the team signs off because perplexity barely moved.
mediumempirical finding - Choose models on a task-specific eval set, not on public leaderboard rank
A team picks the model at the top of a public leaderboard for a booking-change assistant.
easypractitioner heuristic - Treat public benchmark scores as possibly contaminated and keep a private held-out set
A vendor's model posts a high score on a well-known public benchmark, and the team expects similar quality on its own tasks.
mediumempirical finding - Put a confidence interval on every eval score and size the set for the difference you care about
A prompt revision raises accuracy on a 200-item eval set from 80 to 83 percent, and the team announces an improvement.
mediumdocumented fact - Compare two models on the same items with a paired analysis
Model A and model B are run on the same 300 eval items, and their accuracies are compared as two independent proportions.
mediumdocumented fact - Run every pairwise LLM judgment in both orders and count only consistent verdicts
An LLM judge compares old and new assistant answers, always shown in the same order, and prefers the new one 62 percent of the time.
mediumempirical finding - Control for answer length when an LLM judge or reward model scores outputs
A new prompt makes answers 40 percent longer, and the LLM judge's win rate jumps.
mediumempirical finding - Avoid letting a model family judge its own outputs
A team uses the same model to generate answers and to judge them against a competitor's answers.
mediumempirical finding - Validate an LLM judge against human labels before trusting its scores
A team plans to replace human review of complaint replies with an LLM judge to speed up evaluation.
mediumpractitioner heuristic - Use pairwise judging to rank close variants and rubric grading to set pass bars
A team asks an LLM judge to score answers from 1 to 10 to decide between two similar prompt versions.
mediumpractitioner heuristic - Check everything verifiable with code before reaching for an LLM judge
An eval pipeline sends every output to an LLM judge, including checks such as valid JSON, word limits and whether the booking reference matches the input.
easypractitioner heuristic - Turn every production failure into a regression test that gates changes
A model upgrade reintroduces a bug that was fixed months ago, quoting outdated pet-travel rules.
easypractitioner heuristic - Split the eval set into a development set for iteration and a locked test set for decisions
A team iterates on prompts for weeks against one 300-item eval set and reports the final score as expected production quality.
mediumpractitioner heuristic - Calibrate LLM confidence on labelled data before using it as a threshold
A claims-triage assistant auto-approves cases when the model's token probability or stated confidence exceeds 0.9.
hardempirical finding - Locate RLHF failures in feedback, reward model or policy before changing the recipe
After preference tuning, an internal assistant becomes evasive on some topics and overconfident on others, and the team considers switching alignment methods.
hardpractitioner heuristic - Track a gold metric while optimising against a reward model, and stop before it turns down
During RL fine-tuning, the reward-model score keeps rising run after run, and the team keeps training.
hardempirical finding - Choose DPO for simplicity, but check for distribution shift in the preference data
A team with a fixed preference dataset picks DPO because it avoids training a reward model and running reinforcement learning.
hardempirical finding - Measure output diversity after preference tuning when variety is part of the product
A preference-tuned model generates marketing copy for 300 routes, and the drafts start to sound alike.
mediumempirical finding - Check whether a preference-tuning gain is just longer answers
A preference-tuned model wins more human and model comparisons, and its answers are noticeably longer than before.
mediumempirical finding - Re-measure core capabilities after alignment tuning to catch an alignment tax
A model tuned for helpfulness and tone on support conversations is promoted after winning preference comparisons.
mediumempirical finding - Pick a generative model family by the constraint that binds, not by popularity
A retailer wants synthetic product images for layout testing and assumes the newest diffusion model is the only option.
mediumdocumented fact - Sweep guidance strength as a fidelity-versus-diversity knob, like temperature
A team generating synthetic training images sets a high guidance scale because individual samples look sharpest.
mediumempirical finding