Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Check sample ratio mismatch before reading any experiment metric
A 50/50 experiment shows 100,000 users in control and 98,500 in treatment, and the readout meeting is about to start.
easydocumented fact - Pair every target metric with a counter-metric that exposes gaming
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
easypractitioner heuristic - State the base rate before quoting a detector's accuracy to stakeholders
A fraud or no-show alert is described as "95% accurate", and a manager plans staffing around its flags.
easyempirical finding - Write down the grain of every table and assert it before the first aggregate
You start a KPI query on an unfamiliar table, or a table whose definition changed recently.
easydocumented fact - Deduplicate on a business key and keep the latest version by update time, not with DISTINCT
A raw table contains repeated rows for the same flight and date, some identical and some with a revised revenue value.
easypractitioner heuristic - Normalize categorical keys and compare their cardinality with a reference list before grouping
A GROUP BY route returns more distinct routes than the network has, including near-identical codes.
easypractitioner heuristic - Distinguish "zero", "unknown" and "not applicable" instead of letting one NULL mean all three
A cancellations column is NULL for some flights, and analysts disagree on whether that means no cancellations, missing data or a flight that never operated.
easypractitioner heuristic - Encode impossible values as explicit integrity rules that quarantine rows and count them
EDA finds load factors above 1 and negative bookings in a raw table that feeds a forecast.
easypractitioner heuristic - Track distinct-value counts per column as an early warning for upstream changes
A model or dashboard depends on categorical columns fed by systems you do not own.
easypractitioner heuristic - Check source freshness against an explicit SLA, more often than the SLA itself
A morning revenue dashboard occasionally shows yesterday's numbers unchanged because an upstream load failed silently.
easydocumented fact - Check row volume per entity and period to find gaps that schema tests cannot see
A table passes all type and null checks, yet a route's history has a hole of several days.
easypractitioner heuristic - Write chart titles that state the finding and its scope, not the variables plotted
A slide is titled "Bookings by month, 2023-2024" and the presenter explains the point verbally.
easypractitioner heuristic - Skip feature scaling for tree ensembles and spend the effort on features that carry signal
You are about to add StandardScaler or MinMaxScaler in front of a random forest, gradient boosting model or single decision tree.
easydocumented fact - Scale features before k-NN, SVMs and k-means or the largest unit decides the distance
A distance- or kernel-based model uses features in mixed units, such as average fare in currency, load factor in 0-1 and days to departure in days.
easydocumented fact - Use RobustScaler when a few extreme values distort mean-and-variance scaling, then decide about the extremes separately
A scaled feature has a handful of huge values, such as a 10x passenger outlier or a corporate group booking.
easydocumented fact - Put every fitted preprocessing step inside the cross-validated pipeline, not before it
Your notebook imputes, scales or selects features on the whole dataframe, then calls crossvalscore on the result.
easydocumented fact - Group infrequent levels before one-hot encoding a high-cardinality categorical
One-hot encoding a feature such as travel agency or product SKU would create thousands of sparse columns.
easydocumented fact - Do not give a linear model ordinal codes for unordered categories
A nominal feature such as market or payment method is label-encoded as 0, 1, 2 and fed to a linear, kernel or distance model.
easypractitioner heuristic - Let gradient boosting route missing values natively before building an imputer
You plan an imputation step in front of HistGradientBoosting, LightGBM or XGBoost.
easydocumented fact - Replace sentinel values like -999 before they reach linear or distance models
A source system encodes missing or not-applicable as -999, 9999 or 0.
easypractitioner heuristic - Guard ratio features against zero and tiny denominators
You build ratios such as revenue per seat, cancellations per booking or spend per visit.
easypractitioner heuristic - Encode cyclical calendar features so December sits next to January for linear and distance models
Month, weekday or hour enters a linear, kernel or distance model as an integer.
easypractitioner heuristic - Remove features that are recorded because of the outcome, not before it
A candidate feature is filled in by a process that happens after or because of the target event, such as a cancellation reason, refund amount or collections flag.
easydocumented fact - Keep raw identifiers out of the feature set and use them for grouping instead
Customer id, flight number, device id or row index ends up as a model input.
easypractitioner heuristic - Take clipping limits from training-fold quantiles and record them as model parameters
You winsorize heavy-tailed features or targets to limit the influence of extremes.
easypractitioner heuristic - Standardize features before PCA unless they share a meaningful common unit
You run PCA on features measured in different units, such as revenue, seat counts and percentages.
easydocumented fact - Scale sparse matrices without centering them
Your features are a sparse matrix, for example one-hot route-by-channel indicators or TF-IDF text features, and you add StandardScaler.
easydocumented fact - Score a dummy baseline on the same split and metric before reading any model result
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
easydocumented fact - Benchmark against the incumbent business rule, not only against a trivial baseline
A model beats a dummy baseline and you are drafting the launch proposal.
easypractitioner heuristic - Replace accuracy with prevalence-aware metrics when the positive class is rare
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
easydocumented fact - Resample inside the cross-validation pipeline so validation folds keep the real class mix
You undersample or oversample the whole dataset, then cross-validate or split into train and test.
easydocumented fact - Deduplicate at the correct grain before splitting so copies do not straddle train and test
The raw table has exact or near-duplicate rows, for example re-sent records or late updates of the same event.
easydocumented fact - Plot a learning curve before asking for more data
Stakeholders ask whether another year of history or more labelling budget will improve the model.
easydocumented fact - Read a validation curve for one hyperparameter to locate under- and overfitting, then confirm on held-out data
You are unsure whether to increase or decrease tree depth, regularization strength or the number of neighbours.
easydocumented fact - Give early stopping its own validation split and never point it at the test set
You train LightGBM, XGBoost or a neural network with early stopping and pass the test set as the evaluation set.
easydocumented fact - Rank error slices by total cost, not by error rate alone
An error-analysis table lists dozens of segments with their error rates and the team must choose what to fix first.
easypractitioner heuristic - Read the most confident wrong predictions first when debugging a classifier
You start error analysis on a classifier with thousands of misclassifications.
easypractitioner heuristic - Use many rolling origins that span the seasons you care about, not one final holdout
A forecast model is chosen from a single holdout of the last eight weeks before peak season.
easydocumented fact - Fit normalisation statistics at each cutoff, never on the full time series
A notebook standardises the whole series with its mean and standard deviation, then runs a rolling-origin backtest.
easydocumented fact - In direct models, keep every lag at least as long as the horizon it predicts
A direct model for 14-days-ahead demand uses lag 7 sales as a feature.
easypractitioner heuristic - Shift before rolling so window features exclude the target period
A 7-day rolling mean of sales is computed and joined to the same row as the target.
easydocumented fact - Write a data contract for every upstream table a model depends on
A model breaks because an upstream team renamed a status code and nobody told the data science team.
easypractitioner heuristic - Monitor distributions to catch semantic changes that schemas cannot see
A price column switches from cents to currency units in a release; schema, types and null rates are unchanged.
easypractitioner heuristic - Make batch scoring writes idempotent so retries cannot duplicate decisions
A nightly scoring job times out halfway and the scheduler retries it.
easypractitioner heuristic - Keep rollback one alias change away and test it before you need it
A new model version misbehaves, and rollback requires rebuilding the old container and feature pipeline.
easydocumented fact - Record why a model version was promoted, not only that it was
Six months after promotion, nobody can explain why version 14 replaced version 12 or what evidence was used.
easypractitioner heuristic - Instrument a model service with the golden signals plus model-specific signals
A model endpoint has a health check that returns OK, and users report slow and odd recommendations.
easydocumented fact - Check model and dataset licences before a prototype becomes a product
A prototype uses an open-weights model and a public dataset, and the business wants to launch it commercially.
easydocumented fact - Write a datasheet for any dataset other teams will train on
A curated training table is published internally, and three teams start building models on it.
easypractitioner heuristic - Ship a model card with intended use and performance by relevant slice
A claims-triage model is handed to an operations team with only an overall accuracy figure.
easypractitioner heuristic - Use batch scoring when decisions are made on a schedule
A team plans a low-latency online endpoint for a churn model whose scores are used in a weekly retention campaign.
easypractitioner heuristic - Track unit cost per decision, not just the monthly cloud bill
Monthly ML infrastructure cost rose 40%, and nobody can say whether it is growth, waste or a new model.
easypractitioner heuristic - Declare model incidents early and give one person stakeholder communication
A pricing model has produced obviously wrong fares for an hour, and engineers are debugging while managers message them for updates.
easydocumented fact - Run blameless postmortems with triggers defined before incidents happen
After a model rollback, the team discusses who merged the bad change and moves on without writing anything.
easydocumented fact - Monitor default and fallback rates to catch silent model failures
An upstream join breaks, a key feature becomes null, and the pipeline fills it with the training median.
easypractitioner heuristic - Negotiate an ambiguous AI request down to one decision, one metric and a baseline
A director asks the team to 'use AI to optimise the network' within a quarter.
easypractitioner heuristic - Give every production model a named owner, a runbook and a support path
A model alert fires at night, and the person who built it left the company months ago.
easypractitioner heuristic - Classify data-quality checks into blocking and warning tiers before they fire
A new validation suite fails the nightly pipeline on a minor null-rate rise, and the business misses its morning report.
easypractitioner heuristic - Estimate the value of a model with a back-of-envelope calculation before building it
A team proposes a three-month project to improve a no-show prediction model.
easypractitioner heuristic - Keep a decision and assumption log so a model survives handover
A modelling project is handed to a new team, and the reasons for its target definition, exclusions and thresholds exist only in old chat threads.
easypractitioner heuristic - Assert prediction and target shapes right before every loss call
A regression network returns predictions of shape (N, 1) while the target tensor is (N,), for example predicted load factor per flight-day.
easydocumented fact - Count parameters by hand before training and compare with your number of rows
You size a multilayer perceptron for a tabular problem such as predicting final bookings from 20 booking-curve features.
easydocumented fact - Feed raw logits to CrossEntropyLoss, never softmax outputs
A multi-class model ends in nn.Softmax and its output goes straight into nn.CrossEntropyLoss.
easydocumented fact - Use BCEWithLogitsLoss instead of a sigmoid followed by BCELoss
A binary model, such as a no-show or fraud classifier, ends in a sigmoid layer and is trained with BCELoss.
easydocumented fact - Check that the first loss value equals what chance predicts
You start a new classifier training run, for example a 5-class complaint router.
easypractitioner heuristic - Make sure BatchNorm uses running statistics when you evaluate and serve
Validation accuracy differs between two scoring scripts, or a model scores the same image differently in a batch of one and in a full batch.
easydocumented fact - Turn dropout off at inference unless you are deliberately sampling
A deployed demand model returns slightly different predictions for the same request, and nobody changed the code.
easydocumented fact - Overfit a tiny batch before launching full training
A new model or data pipeline is ready and you are about to start a multi-hour training run.
easypractitioner heuristic - Compute input normalization statistics on the training split only
You standardize tabular inputs for a neural net with a mean and standard deviation computed on the full dataset before splitting.
easydocumented fact - Beat a tuned gradient-boosting baseline before shipping a neural net on tabular data
A team proposes a deep model for a tabular problem, such as predicting final bookings per flight from booking-curve, fare and calendar features.
easyempirical finding - Compute CNN feature-map sizes on paper before writing the classifier head
You stack convolution and pooling layers for an image task and hard-code the input size of the first linear layer.
easydocumented fact - Reproduce the pretrained model's preprocessing exactly before fine-tuning
You load a pretrained image or text backbone and feed it data prepared by your own pipeline.
easypractitioner heuristic - Score embeddings with the similarity function the model was trained for
You build semantic search over fare-rule documents and pick cosine similarity by habit for an embedding model you have not read about.
easydocumented fact - Run the first training step on the largest batch and longest sequence you will see
A text model trains with dynamic padding, and batch memory depends on the longest document in each batch.
easypractitioner heuristic - Measure GPU memory with the framework's allocator counters, not nvidia-smi alone
nvidia-smi shows a GPU almost full after training steps, and the team concludes the model cannot fit a larger batch.
easydocumented fact - Profile the input pipeline when GPU utilization stays low
A GPU training job runs slowly and GPU utilization hovers well below full while the CPU is busy.
easypractitioner heuristic - Count tokens per language before sizing prompts, budgets and context limits
An airline assistant must answer customer messages in English, Turkish, Arabic and German, and the team sized its context and cost budget from English test prompts.
easyempirical finding - Supply volatile facts at request time because parametric knowledge is frozen at training
Agents ask a chat model about this season's baggage allowance or a fare rule that changed last month.
easydocumented fact - Match checkpoint type and chat template to the job before judging an open model
An engineer downloads open model weights, sends plain instructions and gets rambling, off-task text.
easydocumented fact - Set decoding parameters per task instead of one global temperature
One shared configuration sets a high temperature for every LLM call, from marketing copy to fare-rule extraction.
easyempirical finding - Repeat evaluation runs because temperature 0 does not guarantee identical outputs
A prompt change is approved because a single evaluation run at temperature 0 scored two points higher than the old prompt.
easydocumented fact - Check the stop reason before parsing, and treat truncated output as a failure
A batch job extracts structured itinerary changes and occasionally receives JSON that ends mid-object.
easydocumented fact - Verify every identifier the model emits against the system of record
An assistant drafts rebooking notes that include flight numbers, fare basis codes and policy section numbers.
easypractitioner heuristic - Write success criteria and test cases before the first prompt revision
A team has rewritten a booking-change prompt eleven times, each time reacting to whichever example last went wrong.
easypractitioner heuristic - Use provider-enforced schemas instead of asking for JSON in the prompt
A pipeline extracts passenger name, flight and requested change from emails by asking the model to reply in JSON, then parses with a try/except.
easydocumented fact - Validate business rules after the schema passes, because schemas only check shape
A structured-output extractor for refund requests returns perfectly valid objects, and the team stops checking them.
easypractitioner heuristic - Give every extracted field an explicit unknown value so the model is not forced to invent
An extraction schema requires a booking reference, a fare class and a travel date for every complaint email.
easypractitioner heuristic - Repair failed outputs with the validator's error message and a retry cap
A structured extraction fails business-rule validation on a few percent of documents, and an engineer adds code that silently coerces bad values into defaults.
easypractitioner heuristic - Separate instructions from pasted data with clear delimiters, but do not call it security
A prompt mixes instructions with a pasted customer email, and the model sometimes follows sentences inside the email as if they were instructions.
easypractitioner heuristic - Evaluate the exact artefact you will serve, not the training checkpoint
A LoRA adapter scores well during training, then is merged into the base weights and quantized for serving without re-evaluation.
easypractitioner heuristic - Choose models on a task-specific eval set, not on public leaderboard rank
A team picks the model at the top of a public leaderboard for a booking-change assistant.
easypractitioner heuristic - Check everything verifiable with code before reaching for an LLM judge
An eval pipeline sends every output to an LLM judge, including checks such as valid JSON, word limits and whether the booking reference matches the input.
easypractitioner heuristic - Turn every production failure into a regression test that gates changes
A model upgrade reintroduces a bug that was fixed months ago, quoting outdated pet-travel rules.
easypractitioner heuristic - Chunk along the document's own structure before tuning chunk size
You are ingesting manuals, policies or contracts that have headings, numbered clauses or tables into a RAG index.
easypractitioner heuristic - Keep a BM25 baseline when queries contain codes, identifiers or rare terms
Users search with fare basis codes, error codes, part numbers or policy clause ids, and the team plans a dense-only retriever.
easyempirical finding - Put the strongest evidence at the start of the context, not buried in the middle
Context assembly concatenates passages in arbitrary order, for example by document id or ingestion time.
easyempirical finding - Deduplicate near-identical chunks before they crowd out other evidence
The corpus contains copies of the same policy in several places: the PDF, an intranet page and an email announcement.
easypractitioner heuristic - Validate citation ids in code before showing a cited answer
Your RAG prompt asks the model to cite sources as [doc-id] markers.
easyempirical finding - Report recall@k at the k you actually pass to the model
A retrieval dashboard shows recall@100, while the generation prompt receives the top 5 chunks.
easydocumented fact - Write the workflow in code when the steps are known, and use the model only at fuzzy steps
A team proposes an autonomous agent for a process whose steps are already documented, such as refund eligibility checks or disruption rebooking.
easypractitioner heuristic - Give every agent run hard budgets and explicit stopping conditions
An agent loop runs "until the task is done" with no limit on iterations, tokens, spend or wall-clock time.
easypractitioner heuristic - Expose only the tools a task needs, as an explicit allow-list per route
One agent configuration registers every available tool (email, database, payments, file system) for all requests.
easydocumented fact - Return structured, classified tool errors so the agent can recover or stop
Tool failures reach the model as raw stack traces or a generic "error occurred".
easypractitioner heuristic - Report steps, tool calls and tokens per task alongside agent success rate
Two agent designs reach similar task success, and the comparison stops there.
easypractitioner heuristic - Write tool descriptions like an interface spec and test them with varied inputs
Tools are registered with one-line descriptions such as "search bookings", and the agent misuses them.
easypractitioner heuristic - Cap and shape tool results before they enter the context window
A tool returns a full API payload, a 2,000-row query result or an entire web page to the agent.
easypractitioner heuristic - Assume the system prompt will leak, so never put secrets or access rules only there
A system prompt contains API keys, internal URLs, discount codes, or the only statement of which users may see what.
easydocumented fact - Encode or parameterise model output before it reaches SQL, HTML or a shell
Generated text is inserted into a web page, a database query, a template or a command line.
easydocumented fact - Run GenAI design reviews against a shared threat checklist, not ad-hoc brainstorming
A GenAI feature reaches design review and the security section says "prompt injection mitigated by system prompt".
easydocumented fact - Set per-user quotas and spend limits as a security control, not only as finance hygiene
A public or partner-facing LLM endpoint has no per-user request, token or cost limits.
easydocumented fact - Branch error handling on the error type, because two 429s can need opposite responses
All non-200 responses from the model API go through one generic "wait and retry" handler.
easydocumented fact - Trace each LLM request as spans per stage with model, tokens and finish reason
A user reports a slow or wrong answer, and logs show only the final response and total latency.
easydocumented fact - Version prompt, retrieval settings and model together as one release
Prompts live in code, retrieval parameters in a config service and the model name in an environment variable, each changed independently.
easypractitioner heuristic - Pin dated model versions and schedule migrations before retirement dates
Production calls a moving model alias, or a pinned model that the provider has deprecated.
easydocumented fact - Send offline workloads such as eval runs and nightly enrichment through a batch API
Golden-set evaluations, document classification backfills or nightly summaries run through the real-time API with custom concurrency code.
easydocumented fact - Spell out the window frame when you use LAST_VALUE or NTH_VALUE
A query takes the "latest fare in the booking window" with LASTVALUE(fare) OVER (PARTITION BY flightno ORDER BY snapshotday) and returns the current row's own fare on every row.
easydocumented fact - A negated filter drops NULL rows, so write the NULL branch explicitly
A query keeps "flights not affected by an event" with WHERE event < 'strike', and the analyst expects every other flight to remain.
easydocumented fact - Do money arithmetic in an exact numeric type and compare floats with a tolerance
A reconciliation sums millions of fare amounts stored as double precision, and a total differs from the ledger by a few cents that move when the query plan changes.
easydocumented fact - Page on actionable conditions only, and keep everything else on the dashboard
A data-platform team routes every failed check, slow query and late table to a chat alert, and the channel is muted within a month.
easydocumented fact - Choose Yeo-Johnson over Box-Cox when a feature can be zero or negative
A PowerTransformer with method box-cox was fitted on fare or revenue columns, and later extracts contain zero fares, refunds or negative adjustments.
easydocumented fact