Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Run experiments in whole weekly cycles so the weekday mix matches across the decision
An experiment reaches its sample size on a Tuesday after starting on a Friday, and the team wants to close it.
easypractitioner heuristic - Check sample ratio mismatch before reading any experiment metric
A 50/50 experiment shows 100,000 users in control and 98,500 in treatment, and the readout meeting is about to start.
easydocumented fact - Treat a p-value just under 0.05 as weak evidence that needs replication, not a verdict
A test ends at p = 0.04 on the primary metric and the readout says "statistically significant, ship it".
easydocumented fact - At very large sample sizes, judge the effect against the action threshold, not the p-value
A test with millions of sessions reports p < 0.0001 for a 0.02% conversion change, or an observational query over a year of bookings finds every coefficient significant.
easydocumented fact - Pair every target metric with a counter-metric that exposes gaming
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
easypractitioner heuristic - State the base rate before quoting a detector's accuracy to stakeholders
A fraud or no-show alert is described as "95% accurate", and a manager plans staffing around its flags.
easyempirical finding - Write down the grain of every table and assert it before the first aggregate
You start a KPI query on an unfamiliar table, or a table whose definition changed recently.
easydocumented fact - Compare row counts before and after every join to catch join explosions early
A query joins a flight-level table to a table with several rows per flight, and the resulting passenger total looks plausible but higher than finance reports.
easypractitioner heuristic - Put filters on the optional table in the ON clause, or your LEFT JOIN becomes an inner join
A query LEFT JOINs flights to promotions to keep all flights, then adds WHERE promo.channel = 'web'.
easydocumented fact - Decide whether NULL means zero before trusting AVG or COUNT(column)
An average fare or average delay is computed with AVG over a column that contains NULLs.
easydocumented fact - Expect SUM over an empty or all-NULL group to return NULL, and handle it at the edge
A dashboard shows blank revenue for a route-day with no sales, and a downstream ratio or growth calculation fails or disappears.
easydocumented fact - Deduplicate on a business key and keep the latest version by update time, not with DISTINCT
A raw table contains repeated rows for the same flight and date, some identical and some with a revised revenue value.
easypractitioner heuristic - Never sum across snapshots of a cumulative measure; pick one snapshot per entity
A booking-curve table stores cumulative bookings at several days before departure, and a query sums cumulativebookings per flight.
easypractitioner heuristic - Normalize categorical keys and compare their cardinality with a reference list before grouping
A GROUP BY route returns more distinct routes than the network has, including near-identical codes.
easypractitioner heuristic - Distinguish "zero", "unknown" and "not applicable" instead of letting one NULL mean all three
A cancellations column is NULL for some flights, and analysts disagree on whether that means no cancellations, missing data or a flight that never operated.
easypractitioner heuristic - Encode impossible values as explicit integrity rules that quarantine rows and count them
EDA finds load factors above 1 and negative bookings in a raw table that feeds a forecast.
easypractitioner heuristic - Track distinct-value counts per column as an early warning for upstream changes
A model or dashboard depends on categorical columns fed by systems you do not own.
easypractitioner heuristic - Check source freshness against an explicit SLA, more often than the SLA itself
A morning revenue dashboard occasionally shows yesterday's numbers unchanged because an upstream load failed silently.
easydocumented fact - Check row volume per entity and period to find gaps that schema tests cannot see
A table passes all type and null checks, yet a route's history has a hole of several days.
easypractitioner heuristic - Reconcile a new analytical table to an independent trusted total before anyone uses it
You have built a new revenue or passengers table for analysis or model training, and it will feed decisions.
easypractitioner heuristic - Compute rate KPIs as a ratio of sums unless you deliberately want the average of unit rates
Network load factor is reported as AVG(loadfactor) across flights, while finance computes passengers divided by seats.
easypractitioner heuristic - Cast to a decimal type before dividing integer counts in SQL
A conversion-rate query divides COUNT of bookings by COUNT of searches and returns 0 for every route.
easydocumented fact - Choose the chart from the comparison the reader must make, not from the data type
You need to show revenue share by fare level across three markets and the default tool suggests three pie charts.
easypractitioner heuristic - Start bar axes at zero, and switch to dots or lines when you need to show small differences
A bar chart of route load factors starts its axis at 75% so the differences look clearer.
easydocumented fact - Replace a spaghetti line chart with small multiples that share the same axes
A chart shows weekly bookings for twelve routes as twelve overlapping coloured lines.
easypractitioner heuristic - Write chart titles that state the finding and its scope, not the variables plotted
A slide is titled "Bookings by month, 2023-2024" and the presenter explains the point verbally.
easypractitioner heuristic - Index series to a common base period instead of using a dual-axis chart
You want to compare growth of revenue (in millions) with passengers (in thousands) on one chart.
easypractitioner heuristic - Use a log scale for ratios and multiplicative growth, and label the original values
A chart compares route revenue that spans from thousands to millions, or shows growth multiples above and below 1.
easypractitioner heuristic - Use RobustScaler when a few extreme values distort mean-and-variance scaling, then decide about the extremes separately
A scaled feature has a handful of huge values, such as a 10x passenger outlier or a corporate group booking.
easydocumented fact - Do not give a linear model ordinal codes for unordered categories
A nominal feature such as market or payment method is label-encoded as 0, 1, 2 and fed to a linear, kernel or distance model.
easypractitioner heuristic - Replace sentinel values like -999 before they reach linear or distance models
A source system encodes missing or not-applicable as -999, 9999 or 0.
easypractitioner heuristic - Guard ratio features against zero and tiny denominators
You build ratios such as revenue per seat, cancellations per booking or spend per visit.
easypractitioner heuristic - Log-transform skewed inputs for linear and distance models, not for trees
A numeric feature spans orders of magnitude, such as corporate account spend or group size.
easypractitioner heuristic - Remove features that are recorded because of the outcome, not before it
A candidate feature is filled in by a process that happens after or because of the target event, such as a cancellation reason, refund amount or collections flag.
easydocumented fact - Score a dummy baseline on the same split and metric before reading any model result
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
easydocumented fact - Benchmark against the incumbent business rule, not only against a trivial baseline
A model beats a dummy baseline and you are drafting the launch proposal.
easypractitioner heuristic - Replace accuracy with prevalence-aware metrics when the positive class is rare
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
easydocumented fact - Retire MAPE when actuals can be zero or near zero
You evaluate demand, sales or bookings where some actuals are zero or tiny, for example small routes or slow SKUs.
easydocumented fact - Pick MAE or RMSE from whether the decision needs the median or the mean
You are choosing a training loss or headline metric for a regression model.
easydocumented fact - Deduplicate at the correct grain before splitting so copies do not straddle train and test
The raw table has exact or near-duplicate rows, for example re-sent records or late updates of the same event.
easydocumented fact - Remember scikit-learn's logistic regression is penalized by default before comparing it with statistics output
Coefficients from scikit-learn's LogisticRegression differ from statsmodels or R on the same data, or are used for inference.
easydocumented fact - Rank error slices by total cost, not by error rate alone
An error-analysis table lists dozens of segments with their error rates and the team must choose what to fix first.
easypractitioner heuristic - Establish that the model predicts well before discussing which features it relies on
An importance plot is requested for a model whose held-out score is close to the baseline.
easydocumented fact - Report error by horizon and evaluate at the horizon the decision uses
A dashboard shows one pooled error for a 1-to-90-day forecast that feeds pricing decisions made 21 days before departure.
easydocumented fact - Compute WAPE at the level where actuals cannot sum to zero
WAPE is reported for every store-SKU-week cell and many cells are blank or infinite.
easypractitioner heuristic - Do not average MAPE across series of very different volumes
A portfolio report averages per-series MAPE, and a few slow movers make the whole forecast look broken.
easydocumented fact - Model one-off events as interventions and keep them out of learned seasonality
A city hosted a single major sports final, and hotel and flight demand spiked for one week.
easydocumented fact - Shift before rolling so window features exclude the target period
A 7-day rolling mean of sales is computed and joined to the same row as the target.
easydocumented fact - Prefer simple models on very short series and say how little the backtest can tell you
A new route has 30 weeks of history, and someone proposes a seasonal ARIMA with many parameters.
easydocumented fact - Measure forecast value added at every step, including human overrides
Planners adjust the statistical forecast every month, and nobody knows whether the adjustments help.
easypractitioner heuristic - Track forecast bias separately from accuracy
A forecast with acceptable MAE consistently runs below actual demand on high-season weeks.
easypractitioner heuristic - Build an explicit time grid so missing periods are not mistaken for zeros or skipped
A route's daily table has a nine-day gap where no rows exist, and the lag-7 feature is computed with row offsets.
easypractitioner heuristic - Write a data contract for every upstream table a model depends on
A model breaks because an upstream team renamed a status code and nobody told the data science team.
easypractitioner heuristic - Monitor distributions to catch semantic changes that schemas cannot see
A price column switches from cents to currency units in a release; schema, types and null rates are unchanged.
easypractitioner heuristic - Write a datasheet for any dataset other teams will train on
A curated training table is published internally, and three teams start building models on it.
easypractitioner heuristic - Track unit cost per decision, not just the monthly cloud bill
Monthly ML infrastructure cost rose 40%, and nobody can say whether it is growth, waste or a new model.
easypractitioner heuristic - Declare model incidents early and give one person stakeholder communication
A pricing model has produced obviously wrong fares for an hour, and engineers are debugging while managers message them for updates.
easydocumented fact - Run blameless postmortems with triggers defined before incidents happen
After a model rollback, the team discusses who merged the bad change and moves on without writing anything.
easydocumented fact - Negotiate an ambiguous AI request down to one decision, one metric and a baseline
A director asks the team to 'use AI to optimise the network' within a quarter.
easypractitioner heuristic - Open an executive memo with the decision, the evidence range and the main limit
A four-page analysis of an overbooking model starts with data sources and methods, and the recommendation is on page four.
easyopinion - Present forecasts to decision makers as ranges tied to the actions they support
Leadership receives a single number for next season's demand and builds the capacity plan on it.
easypractitioner heuristic - Classify data-quality checks into blocking and warning tiers before they fire
A new validation suite fails the nightly pipeline on a minor null-rate rise, and the business misses its morning report.
easypractitioner heuristic - Estimate the value of a model with a back-of-envelope calculation before building it
A team proposes a three-month project to improve a no-show prediction model.
easypractitioner heuristic - Keep a decision and assumption log so a model survives handover
A modelling project is handed to a new team, and the reasons for its target definition, exclusions and thresholds exist only in old chat threads.
easypractitioner heuristic - Start text classification with TF-IDF and a linear model
You need to route airline customer complaints into 8 queues and the first idea is to fine-tune a Transformer.
easyempirical finding - Fit the vectorizer vocabulary and IDF weights on training text only
A notebook calls TfidfVectorizer().fittransform(alltexts) and then splits the matrix into train and test.
easydocumented fact - Report per-class recall and macro averages for long-tail intent classifiers
A banking chatbot intent model reports 94% accuracy over 60 intents, and most traffic falls into five intents.
easypractitioner heuristic - Do not treat a t-SNE or UMAP picture of embeddings as evidence of separation
A slide shows customer-message embeddings in 2D with neat colored clusters and claims the model separates segments.
easydocumented fact - Give every extracted field an explicit unknown value so the model is not forced to invent
An extraction schema requires a booking reference, a fare class and a travel date for every complaint email.
easypractitioner heuristic - Supply context instead of personas, because role prompts do not add knowledge
A prompt starts by telling the model it is a world-class revenue-management expert, and the team expects better pricing analysis as a result.
easyempirical finding - Deduplicate near-identical chunks before they crowd out other evidence
The corpus contains copies of the same policy in several places: the PDF, an intranet page and an email announcement.
easypractitioner heuristic - Label the evidence passages in your golden set, not only the expected answers
You are building the first evaluation set for a document assistant from real user questions.
easypractitioner heuristic - Write down exclusions, covariates and the stopping rule before unblinding, because each free choice is a hidden test
After a flat result, an analyst tries excluding bot-like sessions, adding a covariate, trimming the top 1%, and switching to log revenue, and reports the variant with p = 0.03.
easyempirical finding - Do not read a market-level correlation as a statement about how individual customers behave
Across 60 markets, higher average ancillary spend goes with a higher business-traveller share, and a manager concludes that business travellers spend more on ancillaries.
easydocumented fact - Write the decision table before the result arrives, mapping each interval to an action
A test finishes with a lift of +0.4% and an interval from -0.3% to +1.1%, and the room argues about whether that is a win.
easypractitioner heuristic - Spell out the window frame when you use LAST_VALUE or NTH_VALUE
A query takes the "latest fare in the booking window" with LASTVALUE(fare) OVER (PARTITION BY flightno ORDER BY snapshotday) and returns the current row's own fare on every row.
easydocumented fact - Choose ROW_NUMBER, RANK or DENSE_RANK by deciding what a tie should do
A report asks for the top 3 routes per month by revenue, and two routes tie for third place in some months.
easydocumented fact - Count a right-hand key, not COUNT(*), after a LEFT JOIN
A query LEFT JOINs flights to ancillary sales and reports the number of sales per flight with COUNT(), and flights with no sales show 1.
easydocumented fact - A negated filter drops NULL rows, so write the NULL branch explicitly
A query keeps "flights not affected by an event" with WHERE event < 'strike', and the analyst expects every other flight to remain.
easydocumented fact - In SQLite, treat dates as text with a fixed format and never mix representations in one column
You query the airline sample database, where the date column is TEXT, and a join or filter mixes ISO strings with strings such as 3/4/2023 or with Unix timestamps from another source.
easydocumented fact - Do money arithmetic in an exact numeric type and compare floats with a tolerance
A reconciliation sums millions of fare amounts stored as double precision, and a total differs from the ledger by a few cents that move when the query plan changes.
easydocumented fact - Pair the ISO week number with the ISO week-year, never with the calendar year
A weekly report groups by EXTRACT(year ...) and EXTRACT(week ...), and the last days of December show up as week 1 next to January's data.
easydocumented fact - Plot the rows before trusting a summary table, because identical statistics can hide different shapes
A segment comparison table shows near-identical mean, standard deviation and correlation for two route groups, and the summary is about to be filed as "no difference".
easypractitioner heuristic - Try several histogram bin widths and check density plots for impossible values
A fare distribution is shown as one histogram with software-default bins, and a "second peak" is discussed in a meeting.
easydocumented fact - Replace rainbow palettes and many-color legends with ordered scales and direct labels
A route-by-month heat map uses a rainbow scale, and a line chart of eleven routes needs a legend to read.
easydocumented fact - Page on actionable conditions only, and keep everything else on the dashboard
A data-platform team routes every failed check, slow query and late table to a chat alert, and the channel is muted within a month.
easydocumented fact - Show freshness and metric definitions on the dashboard itself
An executive dashboard shows "bookings today", the upstream load is two hours late, and viewers compare it to a figure from another team that defines a booking differently.
easypractitioner heuristic - Check the realised bin count when quantile-binning a zero-inflated feature
KBinsDiscretizer with strategy quantile and ten bins is applied to a feature with a large spike, for example 60% zeros in days since last cancellation.
easydocumented fact - Size recall and precision intervals by the number of positives, not by the number of rows
A test set has 50,000 rows and 40 positives, and the report quotes recall of 0.80 with no interval.
easypractitioner heuristic - Publish per-class recall with macro and weighted averages when a multiclass problem has rare classes
A multiclass model (disruption cause, claim type, complaint category) is summarized by accuracy or weighted F1, and one costly class holds a few percent of rows.
easydocumented fact - End each backtest origin on a complete aggregation period
Daily bookings are rolled up to weeks, and the cutoff falls on a Wednesday; the training week's total contains three days and the test week begins mid-week.
easypractitioner heuristic - Build the naive benchmark at the production horizon from the origin value, not from the previous day
A 28-day-ahead model is compared with a naive baseline computed as yesterday's actual for every target day.
easydocumented fact - Retire sMAPE from forecast reports because of its asymmetry and its failure at zero
A report uses symmetric MAPE because it is bounded and the team believes it fixes MAPE's bias toward low forecasts.
easydocumented fact - Put a sampling error on measured interval coverage before calling a model miscalibrated
An 80% interval covered 74% of 50 backtest origins and the team wants to retune the model.
easypractitioner heuristic - Report how many series the new model beats the benchmark on, next to the average improvement
A candidate model improves average MASE over 2,000 series by 4% and the team proposes a full cutover.
easypractitioner heuristic - Take the seasonal difference first and let a seasonal-strength measure decide whether you need one
Daily bookings show a strong weekly pattern and a trend, and an analyst applies a first difference and then wonders why the weekly spikes are still there.
easydocumented fact - Let a damped trend compete with a linear trend whenever automatic forecasts run beyond a few periods
A monthly forecast for 1,000 route-level series is produced automatically, and a linear-trend Holt model projects 24 months of continued growth on routes that recently grew.
easydocumented fact - Convert monthly totals to per-day or per-trading-day rates before modelling seasonality
Monthly revenue for a retailer swings partly because months contain different numbers of trading days, and the seasonal model learns a short-February dip and a long-month bump as seasonality.
easydocumented fact - Check that the method tolerates gaps, and model systematic closures with dummies rather than interpolating them
A store's daily sales are missing on public holidays when it is closed, and the pipeline either fails inside an exponential smoothing fit or silently interpolates the closures.
easydocumented fact - Version KPI definitions like code and restate history when they change
Finance changes 'active customer' from 30 to 90 days of activity in a dashboard, and the trend shows a jump in March (hypothetical).
easypractitioner heuristic - Replace probability words with numeric ranges in written risk statements
A risk memo calls a forecast regression 'likely' and a launch delay 'possible', and three readers take away different odds (hypothetical).
easypractitioner heuristic - Quote an automation benefit as residual human workload, not as the share automated
A business case says a triage model will 'automate 80 percent' of 10,000 monthly cases (hypothetical).
easypractitioner heuristic - Walk stakeholders through twenty concrete predictions before debating the metric
A review meeting keeps stalling on whether an F1 of 0.71 is good (hypothetical).
easypractitioner heuristic - Rehearse a handover by letting the new owner run a full cycle while the builder watches
The builder of a monthly forecasting pipeline is moving teams, and a handover document has been written (hypothetical).
easypractitioner heuristic - Quote an error bar that reflects the size of a small test set before comparing networks
You compare two neural classifiers on a hold-out of 120 rows and one is ahead by four points.
easyempirical finding - Print what a stock stop-word list removes before applying it to complaint or sentiment text
A pipeline strips the packaged English stop-word list before TF-IDF on airline complaints because the tutorial did.
easyempirical finding - Normalise locale-specific numbers and dates before the model reads them
A Turkish fare notice lists 1.299,90 TL and 03.04.2025, and the model is asked to compare amounts and count days to departure.
easypractitioner heuristic - Hand a case to a human with a structured summary, evidence and attempted actions, not the transcript
An agent cannot resolve a baggage-delay claim and escalates to a human agent.
easypractitioner heuristic - Tag every model call with feature, tenant tier and prompt version so spend can be attributed
The monthly model invoice jumps 40% and nobody can say which feature, customer segment or release caused it.
easypractitioner heuristic - Capture the human's edit of a model draft as feedback, because the diff shows what was wrong
Agents edit AI-drafted customer replies before sending, and the only feedback collected is a thumbs-up on the draft.
easypractitioner heuristic - Compute distances from coordinates with a great-circle formula in radians, not Euclidean on degrees
A model needs distance between origins and destinations, or a nearest-station feature, from latitude and longitude columns, and the step runs on raw degree values.
easypractitioner heuristic