Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Run experiments in whole weekly cycles so the weekday mix matches across the decision
An experiment reaches its sample size on a Tuesday after starting on a Friday, and the team wants to close it.
easypractitioner heuristic - Check sample ratio mismatch before reading any experiment metric
A 50/50 experiment shows 100,000 users in control and 98,500 in treatment, and the readout meeting is about to start.
easydocumented fact - Treat a p-value just under 0.05 as weak evidence that needs replication, not a verdict
A test ends at p = 0.04 on the primary metric and the readout says "statistically significant, ship it".
easydocumented fact - At very large sample sizes, judge the effect against the action threshold, not the p-value
A test with millions of sessions reports p < 0.0001 for a 0.02% conversion change, or an observational query over a year of bookings finds every coefficient significant.
easydocumented fact - Pair every target metric with a counter-metric that exposes gaming
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
easypractitioner heuristic - State the base rate before quoting a detector's accuracy to stakeholders
A fraud or no-show alert is described as "95% accurate", and a manager plans staffing around its flags.
easyempirical finding - Write down the grain of every table and assert it before the first aggregate
You start a KPI query on an unfamiliar table, or a table whose definition changed recently.
easydocumented fact - Compare row counts before and after every join to catch join explosions early
A query joins a flight-level table to a table with several rows per flight, and the resulting passenger total looks plausible but higher than finance reports.
easypractitioner heuristic - Put filters on the optional table in the ON clause, or your LEFT JOIN becomes an inner join
A query LEFT JOINs flights to promotions to keep all flights, then adds WHERE promo.channel = 'web'.
easydocumented fact - Decide whether NULL means zero before trusting AVG or COUNT(column)
An average fare or average delay is computed with AVG over a column that contains NULLs.
easydocumented fact - Expect SUM over an empty or all-NULL group to return NULL, and handle it at the edge
A dashboard shows blank revenue for a route-day with no sales, and a downstream ratio or growth calculation fails or disappears.
easydocumented fact - Deduplicate on a business key and keep the latest version by update time, not with DISTINCT
A raw table contains repeated rows for the same flight and date, some identical and some with a revised revenue value.
easypractitioner heuristic - Never sum across snapshots of a cumulative measure; pick one snapshot per entity
A booking-curve table stores cumulative bookings at several days before departure, and a query sums cumulativebookings per flight.
easypractitioner heuristic - Normalize categorical keys and compare their cardinality with a reference list before grouping
A GROUP BY route returns more distinct routes than the network has, including near-identical codes.
easypractitioner heuristic - Distinguish "zero", "unknown" and "not applicable" instead of letting one NULL mean all three
A cancellations column is NULL for some flights, and analysts disagree on whether that means no cancellations, missing data or a flight that never operated.
easypractitioner heuristic - Encode impossible values as explicit integrity rules that quarantine rows and count them
EDA finds load factors above 1 and negative bookings in a raw table that feeds a forecast.
easypractitioner heuristic - Track distinct-value counts per column as an early warning for upstream changes
A model or dashboard depends on categorical columns fed by systems you do not own.
easypractitioner heuristic - Check source freshness against an explicit SLA, more often than the SLA itself
A morning revenue dashboard occasionally shows yesterday's numbers unchanged because an upstream load failed silently.
easydocumented fact - Check row volume per entity and period to find gaps that schema tests cannot see
A table passes all type and null checks, yet a route's history has a hole of several days.
easypractitioner heuristic - Reconcile a new analytical table to an independent trusted total before anyone uses it
You have built a new revenue or passengers table for analysis or model training, and it will feed decisions.
easypractitioner heuristic - Compute rate KPIs as a ratio of sums unless you deliberately want the average of unit rates
Network load factor is reported as AVG(loadfactor) across flights, while finance computes passengers divided by seats.
easypractitioner heuristic - Cast to a decimal type before dividing integer counts in SQL
A conversion-rate query divides COUNT of bookings by COUNT of searches and returns 0 for every route.
easydocumented fact - Choose the chart from the comparison the reader must make, not from the data type
You need to show revenue share by fare level across three markets and the default tool suggests three pie charts.
easypractitioner heuristic - Start bar axes at zero, and switch to dots or lines when you need to show small differences
A bar chart of route load factors starts its axis at 75% so the differences look clearer.
easydocumented fact - Replace a spaghetti line chart with small multiples that share the same axes
A chart shows weekly bookings for twelve routes as twelve overlapping coloured lines.
easypractitioner heuristic - Write chart titles that state the finding and its scope, not the variables plotted
A slide is titled "Bookings by month, 2023-2024" and the presenter explains the point verbally.
easypractitioner heuristic - Index series to a common base period instead of using a dual-axis chart
You want to compare growth of revenue (in millions) with passengers (in thousands) on one chart.
easypractitioner heuristic - Use a log scale for ratios and multiplicative growth, and label the original values
A chart compares route revenue that spans from thousands to millions, or shows growth multiples above and below 1.
easypractitioner heuristic - Skip feature scaling for tree ensembles and spend the effort on features that carry signal
You are about to add StandardScaler or MinMaxScaler in front of a random forest, gradient boosting model or single decision tree.
easydocumented fact - Scale features before k-NN, SVMs and k-means or the largest unit decides the distance
A distance- or kernel-based model uses features in mixed units, such as average fare in currency, load factor in 0-1 and days to departure in days.
easydocumented fact - Use RobustScaler when a few extreme values distort mean-and-variance scaling, then decide about the extremes separately
A scaled feature has a handful of huge values, such as a 10x passenger outlier or a corporate group booking.
easydocumented fact - Put every fitted preprocessing step inside the cross-validated pipeline, not before it
Your notebook imputes, scales or selects features on the whole dataframe, then calls crossvalscore on the result.
easydocumented fact - Group infrequent levels before one-hot encoding a high-cardinality categorical
One-hot encoding a feature such as travel agency or product SKU would create thousands of sparse columns.
easydocumented fact - Do not give a linear model ordinal codes for unordered categories
A nominal feature such as market or payment method is label-encoded as 0, 1, 2 and fed to a linear, kernel or distance model.
easypractitioner heuristic - Let gradient boosting route missing values natively before building an imputer
You plan an imputation step in front of HistGradientBoosting, LightGBM or XGBoost.
easydocumented fact - Replace sentinel values like -999 before they reach linear or distance models
A source system encodes missing or not-applicable as -999, 9999 or 0.
easypractitioner heuristic - Guard ratio features against zero and tiny denominators
You build ratios such as revenue per seat, cancellations per booking or spend per visit.
easypractitioner heuristic - Log-transform skewed inputs for linear and distance models, not for trees
A numeric feature spans orders of magnitude, such as corporate account spend or group size.
easypractitioner heuristic - Encode cyclical calendar features so December sits next to January for linear and distance models
Month, weekday or hour enters a linear, kernel or distance model as an integer.
easypractitioner heuristic - Remove features that are recorded because of the outcome, not before it
A candidate feature is filled in by a process that happens after or because of the target event, such as a cancellation reason, refund amount or collections flag.
easydocumented fact - Keep raw identifiers out of the feature set and use them for grouping instead
Customer id, flight number, device id or row index ends up as a model input.
easypractitioner heuristic - Take clipping limits from training-fold quantiles and record them as model parameters
You winsorize heavy-tailed features or targets to limit the influence of extremes.
easypractitioner heuristic - Standardize features before PCA unless they share a meaningful common unit
You run PCA on features measured in different units, such as revenue, seat counts and percentages.
easydocumented fact - Scale sparse matrices without centering them
Your features are a sparse matrix, for example one-hot route-by-channel indicators or TF-IDF text features, and you add StandardScaler.
easydocumented fact - Score a dummy baseline on the same split and metric before reading any model result
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
easydocumented fact - Benchmark against the incumbent business rule, not only against a trivial baseline
A model beats a dummy baseline and you are drafting the launch proposal.
easypractitioner heuristic - Replace accuracy with prevalence-aware metrics when the positive class is rare
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
easydocumented fact - Retire MAPE when actuals can be zero or near zero
You evaluate demand, sales or bookings where some actuals are zero or tiny, for example small routes or slow SKUs.
easydocumented fact - Pick MAE or RMSE from whether the decision needs the median or the mean
You are choosing a training loss or headline metric for a regression model.
easydocumented fact - Resample inside the cross-validation pipeline so validation folds keep the real class mix
You undersample or oversample the whole dataset, then cross-validate or split into train and test.
easydocumented fact - Deduplicate at the correct grain before splitting so copies do not straddle train and test
The raw table has exact or near-duplicate rows, for example re-sent records or late updates of the same event.
easydocumented fact - Plot a learning curve before asking for more data
Stakeholders ask whether another year of history or more labelling budget will improve the model.
easydocumented fact - Read a validation curve for one hyperparameter to locate under- and overfitting, then confirm on held-out data
You are unsure whether to increase or decrease tree depth, regularization strength or the number of neighbours.
easydocumented fact - Give early stopping its own validation split and never point it at the test set
You train LightGBM, XGBoost or a neural network with early stopping and pass the test set as the evaluation set.
easydocumented fact - Remember scikit-learn's logistic regression is penalized by default before comparing it with statistics output
Coefficients from scikit-learn's LogisticRegression differ from statsmodels or R on the same data, or are used for inference.
easydocumented fact - Rank error slices by total cost, not by error rate alone
An error-analysis table lists dozens of segments with their error rates and the team must choose what to fix first.
easypractitioner heuristic - Read the most confident wrong predictions first when debugging a classifier
You start error analysis on a classifier with thousands of misclassifications.
easypractitioner heuristic - Establish that the model predicts well before discussing which features it relies on
An importance plot is requested for a model whose held-out score is close to the baseline.
easydocumented fact - Use many rolling origins that span the seasons you care about, not one final holdout
A forecast model is chosen from a single holdout of the last eight weeks before peak season.
easydocumented fact - Report error by horizon and evaluate at the horizon the decision uses
A dashboard shows one pooled error for a 1-to-90-day forecast that feeds pricing decisions made 21 days before departure.
easydocumented fact - Compute WAPE at the level where actuals cannot sum to zero
WAPE is reported for every store-SKU-week cell and many cells are blank or infinite.
easypractitioner heuristic - Do not average MAPE across series of very different volumes
A portfolio report averages per-series MAPE, and a few slow movers make the whole forecast look broken.
easydocumented fact - Model one-off events as interventions and keep them out of learned seasonality
A city hosted a single major sports final, and hotel and flight demand spiked for one week.
easydocumented fact - Fit normalisation statistics at each cutoff, never on the full time series
A notebook standardises the whole series with its mean and standard deviation, then runs a rolling-origin backtest.
easydocumented fact - In direct models, keep every lag at least as long as the horizon it predicts
A direct model for 14-days-ahead demand uses lag 7 sales as a feature.
easypractitioner heuristic - Shift before rolling so window features exclude the target period
A 7-day rolling mean of sales is computed and joined to the same row as the target.
easydocumented fact - Prefer simple models on very short series and say how little the backtest can tell you
A new route has 30 weeks of history, and someone proposes a seasonal ARIMA with many parameters.
easydocumented fact - Measure forecast value added at every step, including human overrides
Planners adjust the statistical forecast every month, and nobody knows whether the adjustments help.
easypractitioner heuristic - Track forecast bias separately from accuracy
A forecast with acceptable MAE consistently runs below actual demand on high-season weeks.
easypractitioner heuristic - Try a simple average of diverse forecasts before building a model-selection layer
ETS, ARIMA and a gradient-boosting model each win on different series, and the team plans a meta-model to pick a winner per series.
easyempirical finding - Build an explicit time grid so missing periods are not mistaken for zeros or skipped
A route's daily table has a nine-day gap where no rows exist, and the lag-7 feature is computed with row offsets.
easypractitioner heuristic - Write a data contract for every upstream table a model depends on
A model breaks because an upstream team renamed a status code and nobody told the data science team.
easypractitioner heuristic - Monitor distributions to catch semantic changes that schemas cannot see
A price column switches from cents to currency units in a release; schema, types and null rates are unchanged.
easypractitioner heuristic - Make batch scoring writes idempotent so retries cannot duplicate decisions
A nightly scoring job times out halfway and the scheduler retries it.
easypractitioner heuristic - Keep rollback one alias change away and test it before you need it
A new model version misbehaves, and rollback requires rebuilding the old container and feature pipeline.
easydocumented fact - Record why a model version was promoted, not only that it was
Six months after promotion, nobody can explain why version 14 replaced version 12 or what evidence was used.
easypractitioner heuristic - Instrument a model service with the golden signals plus model-specific signals
A model endpoint has a health check that returns OK, and users report slow and odd recommendations.
easydocumented fact - Check model and dataset licences before a prototype becomes a product
A prototype uses an open-weights model and a public dataset, and the business wants to launch it commercially.
easydocumented fact - Write a datasheet for any dataset other teams will train on
A curated training table is published internally, and three teams start building models on it.
easypractitioner heuristic - Ship a model card with intended use and performance by relevant slice
A claims-triage model is handed to an operations team with only an overall accuracy figure.
easypractitioner heuristic - Use batch scoring when decisions are made on a schedule
A team plans a low-latency online endpoint for a churn model whose scores are used in a weekly retention campaign.
easypractitioner heuristic - Track unit cost per decision, not just the monthly cloud bill
Monthly ML infrastructure cost rose 40%, and nobody can say whether it is growth, waste or a new model.
easypractitioner heuristic - Declare model incidents early and give one person stakeholder communication
A pricing model has produced obviously wrong fares for an hour, and engineers are debugging while managers message them for updates.
easydocumented fact - Run blameless postmortems with triggers defined before incidents happen
After a model rollback, the team discusses who merged the bad change and moves on without writing anything.
easydocumented fact - Monitor default and fallback rates to catch silent model failures
An upstream join breaks, a key feature becomes null, and the pipeline fills it with the training median.
easypractitioner heuristic - Negotiate an ambiguous AI request down to one decision, one metric and a baseline
A director asks the team to 'use AI to optimise the network' within a quarter.
easypractitioner heuristic - Open an executive memo with the decision, the evidence range and the main limit
A four-page analysis of an overbooking model starts with data sources and methods, and the recommendation is on page four.
easyopinion - Present forecasts to decision makers as ranges tied to the actions they support
Leadership receives a single number for next season's demand and builds the capacity plan on it.
easypractitioner heuristic - Give every production model a named owner, a runbook and a support path
A model alert fires at night, and the person who built it left the company months ago.
easypractitioner heuristic - Classify data-quality checks into blocking and warning tiers before they fire
A new validation suite fails the nightly pipeline on a minor null-rate rise, and the business misses its morning report.
easypractitioner heuristic - Estimate the value of a model with a back-of-envelope calculation before building it
A team proposes a three-month project to improve a no-show prediction model.
easypractitioner heuristic - Keep a decision and assumption log so a model survives handover
A modelling project is handed to a new team, and the reasons for its target definition, exclusions and thresholds exist only in old chat threads.
easypractitioner heuristic - Assert prediction and target shapes right before every loss call
A regression network returns predictions of shape (N, 1) while the target tensor is (N,), for example predicted load factor per flight-day.
easydocumented fact - Count parameters by hand before training and compare with your number of rows
You size a multilayer perceptron for a tabular problem such as predicting final bookings from 20 booking-curve features.
easydocumented fact - Feed raw logits to CrossEntropyLoss, never softmax outputs
A multi-class model ends in nn.Softmax and its output goes straight into nn.CrossEntropyLoss.
easydocumented fact - Use BCEWithLogitsLoss instead of a sigmoid followed by BCELoss
A binary model, such as a no-show or fraud classifier, ends in a sigmoid layer and is trained with BCELoss.
easydocumented fact - Check that the first loss value equals what chance predicts
You start a new classifier training run, for example a 5-class complaint router.
easypractitioner heuristic - Make sure BatchNorm uses running statistics when you evaluate and serve
Validation accuracy differs between two scoring scripts, or a model scores the same image differently in a batch of one and in a full batch.
easydocumented fact - Turn dropout off at inference unless you are deliberately sampling
A deployed demand model returns slightly different predictions for the same request, and nobody changed the code.
easydocumented fact - Overfit a tiny batch before launching full training
A new model or data pipeline is ready and you are about to start a multi-hour training run.
easypractitioner heuristic - Compute input normalization statistics on the training split only
You standardize tabular inputs for a neural net with a mean and standard deviation computed on the full dataset before splitting.
easydocumented fact - Beat a tuned gradient-boosting baseline before shipping a neural net on tabular data
A team proposes a deep model for a tabular problem, such as predicting final bookings per flight from booking-curve, fare and calendar features.
easyempirical finding - Compute CNN feature-map sizes on paper before writing the classifier head
You stack convolution and pooling layers for an image task and hard-code the input size of the first linear layer.
easydocumented fact - Reproduce the pretrained model's preprocessing exactly before fine-tuning
You load a pretrained image or text backbone and feed it data prepared by your own pipeline.
easypractitioner heuristic - Start text classification with TF-IDF and a linear model
You need to route airline customer complaints into 8 queues and the first idea is to fine-tune a Transformer.
easyempirical finding - Fit the vectorizer vocabulary and IDF weights on training text only
A notebook calls TfidfVectorizer().fittransform(alltexts) and then splits the matrix into train and test.
easydocumented fact - Report per-class recall and macro averages for long-tail intent classifiers
A banking chatbot intent model reports 94% accuracy over 60 intents, and most traffic falls into five intents.
easypractitioner heuristic - Score embeddings with the similarity function the model was trained for
You build semantic search over fare-rule documents and pick cosine similarity by habit for an embedding model you have not read about.
easydocumented fact - Do not treat a t-SNE or UMAP picture of embeddings as evidence of separation
A slide shows customer-message embeddings in 2D with neat colored clusters and claims the model separates segments.
easydocumented fact - Run the first training step on the largest batch and longest sequence you will see
A text model trains with dynamic padding, and batch memory depends on the longest document in each batch.
easypractitioner heuristic - Measure GPU memory with the framework's allocator counters, not nvidia-smi alone
nvidia-smi shows a GPU almost full after training steps, and the team concludes the model cannot fit a larger batch.
easydocumented fact - Profile the input pipeline when GPU utilization stays low
A GPU training job runs slowly and GPU utilization hovers well below full while the CPU is busy.
easypractitioner heuristic - Count tokens per language before sizing prompts, budgets and context limits
An airline assistant must answer customer messages in English, Turkish, Arabic and German, and the team sized its context and cost budget from English test prompts.
easyempirical finding - Supply volatile facts at request time because parametric knowledge is frozen at training
Agents ask a chat model about this season's baggage allowance or a fare rule that changed last month.
easydocumented fact - Match checkpoint type and chat template to the job before judging an open model
An engineer downloads open model weights, sends plain instructions and gets rambling, off-task text.
easydocumented fact - Set decoding parameters per task instead of one global temperature
One shared configuration sets a high temperature for every LLM call, from marketing copy to fare-rule extraction.
easyempirical finding - Repeat evaluation runs because temperature 0 does not guarantee identical outputs
A prompt change is approved because a single evaluation run at temperature 0 scored two points higher than the old prompt.
easydocumented fact - Check the stop reason before parsing, and treat truncated output as a failure
A batch job extracts structured itinerary changes and occasionally receives JSON that ends mid-object.
easydocumented fact - Verify every identifier the model emits against the system of record
An assistant drafts rebooking notes that include flight numbers, fare basis codes and policy section numbers.
easypractitioner heuristic - Write success criteria and test cases before the first prompt revision
A team has rewritten a booking-change prompt eleven times, each time reacting to whichever example last went wrong.
easypractitioner heuristic