Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Size an A/B test from the smallest effect you would act on, not the effect you hope to see
A product owner asks how long to run a booking-page test and offers the uplift they expect, or no target at all.
mediumpractitioner heuristic - Discount significant wins from underpowered tests because their effect sizes are exaggerated
A small or short test returns a statistically significant uplift that is several times larger than past changes of the same kind.
harddocumented fact - Do not stop a fixed-horizon A/B test the first time the dashboard turns significant
Stakeholders check the experiment dashboard daily and ask to ship as soon as the p-value drops below 0.05.
mediumdocumented fact - Run experiments in whole weekly cycles so the weekday mix matches across the decision
An experiment reaches its sample size on a Tuesday after starting on a Friday, and the team wants to close it.
easypractitioner heuristic - Name one primary metric and correct for multiplicity across the rest of the scorecard
An experiment scorecard shows 20 or more metrics and the readout highlights whichever ones moved.
mediumdocumented fact - Treat a winning segment found after a flat test as a hypothesis for the next test
The overall result is flat, so the analyst slices by country, device, fare class and tenure until one subgroup shows a significant uplift.
mediumdocumented fact - Check sample ratio mismatch before reading any experiment metric
A 50/50 experiment shows 100,000 users in control and 98,500 in treatment, and the readout meeting is about to start.
easydocumented fact - Only segment experiment results by attributes the treatment cannot change
A readout splits users by "saw the new fare-comparison widget", "completed search" or another behaviour measured after assignment.
harddocumented fact - Plot the treatment effect by day of exposure before trusting a short-term uplift
A new seat-map design or prompt shows a strong uplift in a two-week test and the team wants to extrapolate it to the annual plan.
mediumdocumented fact - Use a pre-period covariate (CUPED-style adjustment) to shrink variance before buying more traffic
An experiment on revenue per user needs more weeks than the roadmap allows, and the metric has a strong user-level history.
hardempirical finding - Match the variance formula to the randomization unit when metrics are computed per page or per session
Users are randomized, but the metric is click-through per page view or conversion per search, and the analysis treats each page as an independent observation.
harddocumented fact - Report the numerator and denominator next to every ratio metric in an experiment
Click-through rate or revenue per search improves in treatment, and nobody has looked at clicks and searches separately.
mediumdocumented fact - Communicate a null result with the effects it rules out, not as "no impact"
A well-run test of a new upsell banner ends with p = 0.4 and a stakeholder asks whether the feature "does nothing".
mediumdocumented fact - Power guardrail metrics for the harm you must exclude, or the guardrail cannot protect you
A launch review says "no significant change in cancellations or page latency" based on a test sized only for the primary metric.
mediumdocumented fact - Treat a p-value just under 0.05 as weak evidence that needs replication, not a verdict
A test ends at p = 0.04 on the primary metric and the readout says "statistically significant, ship it".
easydocumented fact - At very large sample sizes, judge the effect against the action threshold, not the p-value
A test with millions of sessions reports p < 0.0001 for a 0.02% conversion change, or an observational query over a year of bookings finds every coefficient significant.
easydocumented fact - Run A/A tests on a new experimentation pipeline before trusting its first A/B result
A team has built its own assignment service and analysis notebook and is about to run its first real experiment.
mediumpractitioner heuristic - Decide how to treat revenue outliers before the test starts, then apply it to both arms
Revenue per user in an experiment swings because a few corporate group bookings land in one arm.
mediumdocumented fact - When the total and every segment move in opposite directions, decompose rate and mix before concluding
Network load factor rose year on year while load factor fell on every individual route.
mediumpractitioner heuristic - Ask what removed units from the data before analysing the survivors
An analysis of "routes we still operate" or "customers active today" is used to judge which strategies work.
mediumpractitioner heuristic - Do not filter or control on a variable caused by both the exposure and the outcome
An analyst studies only customers who filed a complaint, or only flights that sold out, and finds a surprising association between two drivers.
harddocumented fact - Never read price elasticity off observational fares and bookings when prices respond to demand
A regression of bookings on average fare across flights shows a positive slope, and someone proposes raising fares to sell more seats.
mediumpractitioner heuristic - Leave mediators out of the model when you want the total effect of a decision
To estimate the effect of a loyalty email on spend, the analyst adds "sessions after the email" as a control variable.
mediumpractitioner heuristic - Treat a clean pre-trend plot as weak support for parallel trends, not proof
A difference-in-differences analysis of a new airport lounge on one hub shows no significant pre-period difference in trends against other hubs, and the report calls parallel trends "confirmed".
harddocumented fact - With staggered rollouts, do not rely on a single two-way fixed-effects DiD coefficient
A self-service kiosk was rolled out to airports at different months, and the team estimates one coefficient with airport and month fixed effects.
harddocumented fact - Model seasonality explicitly in an interrupted time series or the season will pose as the effect
A new ancillary bundle launched in May, and a before/after regression on monthly revenue per passenger shows a jump.
mediumdocumented fact - Look for other events at the intervention date before attributing a break to your change
Fares were changed on a route group in March, and an interrupted time series shows a sharp drop in load factor in the same month.
mediumdocumented fact - Show the synthetic control's pre-period fit and a backdated placebo before showing the effect
A synthetic control built from other airports estimates that a new hub incentive raised passenger numbers, and the deck shows only the post-launch gap.
harddocumented fact - Curate a small, relevant donor pool for synthetic control instead of including every available unit
A synthetic control for one route uses all 200 other routes as donors and only 12 months of pre-period data.
harddocumented fact - Read "X Granger-causes Y" as "X helps forecast Y", never as "changing X changes Y"
A report claims competitor fares Granger-cause your bookings and proposes reacting to every competitor change.
mediumdocumented fact - Expect the worst performers to improve without treatment and keep an untreated comparison
Operations selects the ten routes with the worst on-time performance last quarter for a new turnaround process, and reports their improvement next quarter.
mediumdocumented fact - Pair every target metric with a counter-metric that exposes gaming
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
easypractitioner heuristic - Refuse to answer a "what if we change X" question with a forecast model
An accurate demand forecast uses fare as a feature, and a manager asks it to predict bookings after a 5% fare cut.
mediumpractitioner heuristic - State the base rate before quoting a detector's accuracy to stakeholders
A fraud or no-show alert is described as "95% accurate", and a manager plans staffing around its flags.
easyempirical finding - Do not rank small groups by raw rates; show intervals or shrink toward the overall rate
A league table ranks sales agents, routes or stores by conversion rate, and the top and bottom places go to units with very few observations.
mediumpractitioner heuristic - Write down the grain of every table and assert it before the first aggregate
You start a KPI query on an unfamiliar table, or a table whose definition changed recently.
easydocumented fact - Compare row counts before and after every join to catch join explosions early
A query joins a flight-level table to a table with several rows per flight, and the resulting passenger total looks plausible but higher than finance reports.
easypractitioner heuristic - Aggregate each fact table to a shared grain before joining two facts together
You need bookings and ancillary revenue per flight, and both live in tables with many rows per flight.
mediumpractitioner heuristic - Put filters on the optional table in the ON clause, or your LEFT JOIN becomes an inner join
A query LEFT JOINs flights to promotions to keep all flights, then adds WHERE promo.channel = 'web'.
easydocumented fact - Decide whether NULL means zero before trusting AVG or COUNT(column)
An average fare or average delay is computed with AVG over a column that contains NULLs.
easydocumented fact - Expect SUM over an empty or all-NULL group to return NULL, and handle it at the edge
A dashboard shows blank revenue for a route-day with no sales, and a downstream ratio or growth calculation fails or disappears.
easydocumented fact - Use NOT EXISTS instead of NOT IN when the subquery column can contain NULL
A query finds customers who never booked with WHERE customerid NOT IN (SELECT customerid FROM bookings), and returns zero rows.
mediumdocumented fact - Specify ROWS and a unique ORDER BY in running totals, because the default frame groups ties
A running total of bookings uses SUM(...) OVER (ORDER BY bookingdate) and several rows share the same date.
mediumdocumented fact - Densify dates before a ROWS-based rolling window, because "7 rows" is not "7 days" when days are missing
A 7-day rolling average of passengers uses AVG(...) OVER (PARTITION BY route ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) on a table with gaps.
mediumdocumented fact - Partition lag features by entity and shift them by the forecast horizon, not by one row
A training table adds LAG(passengers) OVER (ORDER BY date) as a feature for a model that forecasts 14 days ahead.
mediumpractitioner heuristic - Report business metrics by event time and pipeline health by processing time
A dashboard counts bookings by the timestamp at which records landed in the warehouse.
mediumdocumented fact - Mark recent periods as provisional until late-arriving data has had time to land
Yesterday's bookings look 15% low on the morning dashboard, and someone escalates a demand drop.
mediumdocumented fact - Make every load and backfill idempotent so a rerun cannot double-count
A failed nightly job is rerun manually, or a month of history is backfilled after a logic fix.
mediumdocumented fact - Deduplicate on a business key and keep the latest version by update time, not with DISTINCT
A raw table contains repeated rows for the same flight and date, some identical and some with a revised revenue value.
easypractitioner heuristic - Join facts to the dimension version that was valid at event time, not to current attributes
A route was reclassified from domestic to intl-short last year, and a revenue-by-market report joins all history to the current route table.
mediumdocumented fact - Keep facts whose dimension record has not arrived yet, using an explicit unknown member
Bookings for a newly opened route arrive before the route appears in the route dimension, and the reporting query uses an inner join.
mediumdocumented fact - Build training rows with as-of joins so each feature uses only values known at the cutoff
A training table joins each departure to the latest customer and route features in the warehouse.
mediumdocumented fact - Never sum across snapshots of a cumulative measure; pick one snapshot per entity
A booking-curve table stores cumulative bookings at several days before departure, and a query sums cumulativebookings per flight.
easypractitioner heuristic - Store instants in UTC and assign the business date with the local zone of where it happened
Bookings from several countries are aggregated to daily totals using the server's clock or a UTC date.
mediumdocumented fact - Expect 23-hour and 25-hour days in hourly data around daylight-saving changes
An hourly demand or call-volume series in local time shows a missing hour in spring and a doubled hour in autumn.
mediumdocumented fact - Normalize categorical keys and compare their cardinality with a reference list before grouping
A GROUP BY route returns more distinct routes than the network has, including near-identical codes.
easypractitioner heuristic - Check whether missingness depends on other columns before dropping incomplete rows
An analysis drops rows with missing fare or revenue because "only 3% are missing".
mediumdocumented fact - Distinguish "zero", "unknown" and "not applicable" instead of letting one NULL mean all three
A cancellations column is NULL for some flights, and analysts disagree on whether that means no cancellations, missing data or a flight that never operated.
easypractitioner heuristic - Classify an outlier as error, rare truth or regime change before deciding what to do with it
One flight shows ten times its usual passengers, or one day's sales are far above every previous day.
mediumdocumented fact - Encode impossible values as explicit integrity rules that quarantine rows and count them
EDA finds load factors above 1 and negative bookings in a raw table that feeds a forecast.
easypractitioner heuristic - Track distinct-value counts per column as an early warning for upstream changes
A model or dashboard depends on categorical columns fed by systems you do not own.
easypractitioner heuristic - Agree a data contract at the producer boundary before building on a shared table
Your forecast or KPI starts consuming a table owned by another team, with no written agreement about it.
mediumpractitioner heuristic - Check source freshness against an explicit SLA, more often than the SLA itself
A morning revenue dashboard occasionally shows yesterday's numbers unchanged because an upstream load failed silently.
easydocumented fact - Check row volume per entity and period to find gaps that schema tests cannot see
A table passes all type and null checks, yet a route's history has a hole of several days.
easypractitioner heuristic - Reconcile a new analytical table to an independent trusted total before anyone uses it
You have built a new revenue or passengers table for analysis or model training, and it will feed decisions.
easypractitioner heuristic - Compute rate KPIs as a ratio of sums unless you deliberately want the average of unit rates
Network load factor is reported as AVG(loadfactor) across flights, while finance computes passengers divided by seats.
easypractitioner heuristic - Cast to a decimal type before dividing integer counts in SQL
A conversion-rate query divides COUNT of bookings by COUNT of searches and returns 0 for every route.
easydocumented fact - Choose the chart from the comparison the reader must make, not from the data type
You need to show revenue share by fare level across three markets and the default tool suggests three pie charts.
easypractitioner heuristic - Start bar axes at zero, and switch to dots or lines when you need to show small differences
A bar chart of route load factors starts its axis at 75% so the differences look clearer.
easydocumented fact - Replace a spaghetti line chart with small multiples that share the same axes
A chart shows weekly bookings for twelve routes as twelve overlapping coloured lines.
easypractitioner heuristic - Write chart titles that state the finding and its scope, not the variables plotted
A slide is titled "Bookings by month, 2023-2024" and the presenter explains the point verbally.
easypractitioner heuristic - Index series to a common base period instead of using a dual-axis chart
You want to compare growth of revenue (in millions) with passengers (in thousands) on one chart.
easypractitioner heuristic - Use a log scale for ratios and multiplicative growth, and label the original values
A chart compares route revenue that spans from thousands to millions, or shows growth multiples above and below 1.
easypractitioner heuristic - Flag capacity-constrained observations before treating sales as demand
A forecast or price analysis uses passengers or units sold as the target, and some flights sold out or some stores ran out of stock.
mediumempirical finding - Decompose a revenue change into volume, price and mix before explaining it
Quarterly revenue rose 8% and leadership asks why, while passengers, average fares and the route mix all changed.
mediumpractitioner heuristic - Skip feature scaling for tree ensembles and spend the effort on features that carry signal
You are about to add StandardScaler or MinMaxScaler in front of a random forest, gradient boosting model or single decision tree.
easydocumented fact - Scale features before k-NN, SVMs and k-means or the largest unit decides the distance
A distance- or kernel-based model uses features in mixed units, such as average fare in currency, load factor in 0-1 and days to departure in days.
easydocumented fact - Standardize predictors before ridge, lasso or elastic net so the penalty is unit-free
You fit a penalized linear or logistic model where features are measured in different units.
mediumdocumented fact - Use RobustScaler when a few extreme values distort mean-and-variance scaling, then decide about the extremes separately
A scaled feature has a handful of huge values, such as a 10x passenger outlier or a corporate group booking.
easydocumented fact - Put every fitted preprocessing step inside the cross-validated pipeline, not before it
Your notebook imputes, scales or selects features on the whole dataframe, then calls crossvalscore on the result.
easydocumented fact - Make target encoding fold-safe with cross fitting; fit().transform() on training rows leaks
You replace a categorical feature such as route or sales channel with the mean target per category.
mediumdocumented fact - Shrink target encodings of rare categories toward the global mean
Many categories have only a handful of rows, for example new routes or long-tail products.
mediumdocumented fact - Consider CatBoost's ordered target statistics when categorical encoding is the main leakage risk
A tabular problem is dominated by high-cardinality categoricals and your hand-built target encoding keeps leaking or overfitting.
mediumdocumented fact - Group infrequent levels before one-hot encoding a high-cardinality categorical
One-hot encoding a feature such as travel agency or product SKU would create thousands of sparse columns.
easydocumented fact - Decide in advance how a model treats categories that first appear after deployment
A categorical feature grows over time: new routes, new products, new store codes.
mediumdocumented fact - Regularize LightGBM's native categorical splits when categories are many and small
You pass a high-cardinality integer-coded categorical to LightGBM as categoricalfeature.
mediumdocumented fact - Do not give a linear model ordinal codes for unordered categories
A nominal feature such as market or payment method is label-encoded as 0, 1, 2 and fed to a linear, kernel or distance model.
easypractitioner heuristic - Add a missing indicator when the fact of missingness may carry signal
A feature is missing more often for one segment or outcome, such as revenue missing more on intl-long flights in the synthetic raw table.
mediumdocumented fact - Let gradient boosting route missing values natively before building an imputer
You plan an imputation step in front of HistGradientBoosting, LightGBM or XGBoost.
easydocumented fact - Avoid single mean imputation when the analysis is inferential rather than predictive
A stakeholder wants coefficients, confidence intervals or group comparisons from data with substantial missingness.
mediumdocumented fact - Replace sentinel values like -999 before they reach linear or distance models
A source system encodes missing or not-applicable as -999, 9999 or 0.
easypractitioner heuristic - Monitor missing-rate shifts because a changed form or feed silently rewrites learned missingness
A model uses missing indicators or native NaN routing, and an upstream form, API or ETL job changes.
mediumpractitioner heuristic - Add explicit interaction terms to a linear model when the effect of one driver depends on another
Domain knowledge says an effect varies by context, for example price sensitivity that differs between short-haul and long-haul markets.
mediumpractitioner heuristic - Cap polynomial expansion degree and regularize, because generated features grow combinatorially
You apply PolynomialFeatures of degree 3 or more to dozens of inputs to capture nonlinearity.
mediumpractitioner heuristic - Guard ratio features against zero and tiny denominators
You build ratios such as revenue per seat, cancellations per booking or spend per visit.
easypractitioner heuristic - When you log-transform a skewed target, correct the back-transformed predictions toward the mean
You model log(revenue) or log(demand) and exponentiate predictions to report totals.
mediumdocumented fact - Log-transform skewed inputs for linear and distance models, not for trees
A numeric feature spans orders of magnitude, such as corporate account spend or group size.
easypractitioner heuristic - Prefer splines to coarse bins when a linear model needs a nonlinear single-feature effect
You plan to cut days to departure or customer age into a few buckets so a linear model can bend.
mediumpractitioner heuristic - Encode cyclical calendar features so December sits next to January for linear and distance models
Month, weekday or hour enters a linear, kernel or distance model as an integer.
easypractitioner heuristic - Express time features relative to the decision moment, not as absolute dates
A model uses raw dates or timestamps as features, for example booking date or departure date as ordinal numbers.
mediumpractitioner heuristic - Compute entity history features as of the prediction timestamp with point-in-time joins
You add aggregates such as a customer's booking count, a route's average load factor or an account's late payments to each training row.
mediumdocumented fact - Run feature selection inside each training fold, never on the full dataset first
You screen hundreds of candidate features by correlation with the target or by model importance, then cross-validate the shortlist.
mediumdocumented fact - Remove features that are recorded because of the outcome, not before it
A candidate feature is filled in by a process that happens after or because of the target event, such as a cancellation reason, refund amount or collections flag.
easydocumented fact - Keep raw identifiers out of the feature set and use them for grouping instead
Customer id, flight number, device id or row index ends up as a model input.
easypractitioner heuristic - Normalize count and frequency encodings by window length so they survive changes in data volume
You encode a category by how often it appears, or an entity by its raw event count.
mediumpractitioner heuristic - Ship the fitted preprocessing with the model so training and serving transform rows identically
Preprocessing lives in a notebook while serving engineers reimplement it in another service or language.
mediumdocumented fact - Take clipping limits from training-fold quantiles and record them as model parameters
You winsorize heavy-tailed features or targets to limit the influence of extremes.
easypractitioner heuristic - Standardize features before PCA unless they share a meaningful common unit
You run PCA on features measured in different units, such as revenue, seat counts and percentages.
easydocumented fact - Drop one dummy level for unpenalized linear regression, keep all levels for penalized models and trees
You one-hot encode categoricals for a linear model and must decide whether to set drop='first'.
mediumdocumented fact - Scale sparse matrices without centering them
Your features are a sparse matrix, for example one-hot route-by-channel indicators or TF-IDF text features, and you add StandardScaler.
easydocumented fact - Score a dummy baseline on the same split and metric before reading any model result
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
easydocumented fact - Benchmark against the incumbent business rule, not only against a trivial baseline
A model beats a dummy baseline and you are drafting the launch proposal.
easypractitioner heuristic - Replace accuracy with prevalence-aware metrics when the positive class is rare
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
easydocumented fact - Do not let a high ROC-AUC on rare events hide poor precision at the operating point
A model for a 0.5% event reports ROC-AUC above 0.9 and the team concludes it is ready.
mediumempirical finding - Read PR-AUC against its prevalence baseline and never compare it across populations with different base rates
You compare PR-AUC of a churn model across countries, months or customer tiers.
mediumdocumented fact - Do not compare R² across test sets with different target variance
A model's R² is 0.85 on the network-wide test set and 0.40 on one route, or differs between two quarters.
mediumdocumented fact - Treat a negative test R² as a signal of shift or broken features, not as a rounding curiosity
A regression model that scored well in validation shows negative R² on a new test period or segment.
mediumdocumented fact - Retire MAPE when actuals can be zero or near zero
You evaluate demand, sales or bookings where some actuals are zero or tiny, for example small routes or slow SKUs.
easydocumented fact - Expect MAPE-selected models to under-forecast and check bias before adopting them
You choose between models or tune a model by minimizing MAPE on positive targets.
mediumpractitioner heuristic - Pick MAE or RMSE from whether the decision needs the median or the mean
You are choosing a training loss or headline metric for a regression model.
easydocumented fact