Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Size an A/B test from the smallest effect you would act on, not the effect you hope to see
A product owner asks how long to run a booking-page test and offers the uplift they expect, or no target at all.
mediumpractitioner heuristic - Do not stop a fixed-horizon A/B test the first time the dashboard turns significant
Stakeholders check the experiment dashboard daily and ask to ship as soon as the p-value drops below 0.05.
mediumdocumented fact - Name one primary metric and correct for multiplicity across the rest of the scorecard
An experiment scorecard shows 20 or more metrics and the readout highlights whichever ones moved.
mediumdocumented fact - Treat a winning segment found after a flat test as a hypothesis for the next test
The overall result is flat, so the analyst slices by country, device, fare class and tenure until one subgroup shows a significant uplift.
mediumdocumented fact - Plot the treatment effect by day of exposure before trusting a short-term uplift
A new seat-map design or prompt shows a strong uplift in a two-week test and the team wants to extrapolate it to the annual plan.
mediumdocumented fact - Report the numerator and denominator next to every ratio metric in an experiment
Click-through rate or revenue per search improves in treatment, and nobody has looked at clicks and searches separately.
mediumdocumented fact - Communicate a null result with the effects it rules out, not as "no impact"
A well-run test of a new upsell banner ends with p = 0.4 and a stakeholder asks whether the feature "does nothing".
mediumdocumented fact - Power guardrail metrics for the harm you must exclude, or the guardrail cannot protect you
A launch review says "no significant change in cancellations or page latency" based on a test sized only for the primary metric.
mediumdocumented fact - Decide how to treat revenue outliers before the test starts, then apply it to both arms
Revenue per user in an experiment swings because a few corporate group bookings land in one arm.
mediumdocumented fact - When the total and every segment move in opposite directions, decompose rate and mix before concluding
Network load factor rose year on year while load factor fell on every individual route.
mediumpractitioner heuristic - Ask what removed units from the data before analysing the survivors
An analysis of "routes we still operate" or "customers active today" is used to judge which strategies work.
mediumpractitioner heuristic - Never read price elasticity off observational fares and bookings when prices respond to demand
A regression of bookings on average fare across flights shows a positive slope, and someone proposes raising fares to sell more seats.
mediumpractitioner heuristic - Leave mediators out of the model when you want the total effect of a decision
To estimate the effect of a loyalty email on spend, the analyst adds "sessions after the email" as a control variable.
mediumpractitioner heuristic - Model seasonality explicitly in an interrupted time series or the season will pose as the effect
A new ancillary bundle launched in May, and a before/after regression on monthly revenue per passenger shows a jump.
mediumdocumented fact - Look for other events at the intervention date before attributing a break to your change
Fares were changed on a route group in March, and an interrupted time series shows a sharp drop in load factor in the same month.
mediumdocumented fact - Read "X Granger-causes Y" as "X helps forecast Y", never as "changing X changes Y"
A report claims competitor fares Granger-cause your bookings and proposes reacting to every competitor change.
mediumdocumented fact - Expect the worst performers to improve without treatment and keep an untreated comparison
Operations selects the ten routes with the worst on-time performance last quarter for a new turnaround process, and reports their improvement next quarter.
mediumdocumented fact - Refuse to answer a "what if we change X" question with a forecast model
An accurate demand forecast uses fare as a feature, and a manager asks it to predict bookings after a 5% fare cut.
mediumpractitioner heuristic - Do not rank small groups by raw rates; show intervals or shrink toward the overall rate
A league table ranks sales agents, routes or stores by conversion rate, and the top and bottom places go to units with very few observations.
mediumpractitioner heuristic - Aggregate each fact table to a shared grain before joining two facts together
You need bookings and ancillary revenue per flight, and both live in tables with many rows per flight.
mediumpractitioner heuristic - Use NOT EXISTS instead of NOT IN when the subquery column can contain NULL
A query finds customers who never booked with WHERE customerid NOT IN (SELECT customerid FROM bookings), and returns zero rows.
mediumdocumented fact - Specify ROWS and a unique ORDER BY in running totals, because the default frame groups ties
A running total of bookings uses SUM(...) OVER (ORDER BY bookingdate) and several rows share the same date.
mediumdocumented fact - Densify dates before a ROWS-based rolling window, because "7 rows" is not "7 days" when days are missing
A 7-day rolling average of passengers uses AVG(...) OVER (PARTITION BY route ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) on a table with gaps.
mediumdocumented fact - Partition lag features by entity and shift them by the forecast horizon, not by one row
A training table adds LAG(passengers) OVER (ORDER BY date) as a feature for a model that forecasts 14 days ahead.
mediumpractitioner heuristic - Report business metrics by event time and pipeline health by processing time
A dashboard counts bookings by the timestamp at which records landed in the warehouse.
mediumdocumented fact - Mark recent periods as provisional until late-arriving data has had time to land
Yesterday's bookings look 15% low on the morning dashboard, and someone escalates a demand drop.
mediumdocumented fact - Make every load and backfill idempotent so a rerun cannot double-count
A failed nightly job is rerun manually, or a month of history is backfilled after a logic fix.
mediumdocumented fact - Join facts to the dimension version that was valid at event time, not to current attributes
A route was reclassified from domestic to intl-short last year, and a revenue-by-market report joins all history to the current route table.
mediumdocumented fact - Keep facts whose dimension record has not arrived yet, using an explicit unknown member
Bookings for a newly opened route arrive before the route appears in the route dimension, and the reporting query uses an inner join.
mediumdocumented fact - Store instants in UTC and assign the business date with the local zone of where it happened
Bookings from several countries are aggregated to daily totals using the server's clock or a UTC date.
mediumdocumented fact - Expect 23-hour and 25-hour days in hourly data around daylight-saving changes
An hourly demand or call-volume series in local time shows a missing hour in spring and a doubled hour in autumn.
mediumdocumented fact - Check whether missingness depends on other columns before dropping incomplete rows
An analysis drops rows with missing fare or revenue because "only 3% are missing".
mediumdocumented fact - Classify an outlier as error, rare truth or regime change before deciding what to do with it
One flight shows ten times its usual passengers, or one day's sales are far above every previous day.
mediumdocumented fact - Agree a data contract at the producer boundary before building on a shared table
Your forecast or KPI starts consuming a table owned by another team, with no written agreement about it.
mediumpractitioner heuristic - Flag capacity-constrained observations before treating sales as demand
A forecast or price analysis uses passengers or units sold as the target, and some flights sold out or some stores ran out of stock.
mediumempirical finding - Decompose a revenue change into volume, price and mix before explaining it
Quarterly revenue rose 8% and leadership asks why, while passengers, average fares and the route mix all changed.
mediumpractitioner heuristic - Avoid single mean imputation when the analysis is inferential rather than predictive
A stakeholder wants coefficients, confidence intervals or group comparisons from data with substantial missingness.
mediumdocumented fact - Add explicit interaction terms to a linear model when the effect of one driver depends on another
Domain knowledge says an effect varies by context, for example price sensitivity that differs between short-haul and long-haul markets.
mediumpractitioner heuristic - Prefer splines to coarse bins when a linear model needs a nonlinear single-feature effect
You plan to cut days to departure or customer age into a few buckets so a linear model can bend.
mediumpractitioner heuristic - Compute entity history features as of the prediction timestamp with point-in-time joins
You add aggregates such as a customer's booking count, a route's average load factor or an account's late payments to each training row.
mediumdocumented fact - Drop one dummy level for unpenalized linear regression, keep all levels for penalized models and trees
You one-hot encode categoricals for a linear model and must decide whether to set drop='first'.
mediumdocumented fact - Read PR-AUC against its prevalence baseline and never compare it across populations with different base rates
You compare PR-AUC of a churn model across countries, months or customer tiers.
mediumdocumented fact - Do not compare R² across test sets with different target variance
A model's R² is 0.85 on the network-wide test set and 0.40 on one route, or differs between two quarters.
mediumdocumented fact - Expect MAPE-selected models to under-forecast and check bias before adopting them
You choose between models or tune a model by minimizing MAPE on positive targets.
mediumpractitioner heuristic - Derive the decision threshold from the cost matrix instead of using 0.5
A calibrated classifier will trigger an action (block a transaction, call a customer, deny an overbooking) and someone asks what cut-off to use.
mediumdocumented fact - When reviewers can handle only K cases, rank and evaluate precision at K instead of choosing a probability cut-off
An operations team can investigate a fixed number of alerts per day regardless of how many the model flags.
mediumpractitioner heuristic - Do not read lasso's choice among correlated features as evidence that the others do not matter
Lasso keeps one of several correlated features (competitor price, fare level, season) and zeroes the rest; someone proposes dropping them from the data.
mediumdocumented fact - Interpret predictions, not individual coefficients, when features are strongly collinear
Coefficients change sign or size when a correlated feature is added, removed or the data period shifts.
mediumdocumented fact - Treat a funnel-shaped residual plot as a modelling decision, not cosmetic noise
Residuals versus fitted values widen as predictions grow, common for revenue, demand and claim size.
mediumdocumented fact - Plot residuals in time order even for models that are not forecasts
A cross-sectional regression is trained on data collected over months, such as flight-level revenue or store-level sales.
mediumdocumented fact - Refuse to turn SHAP values into intervention advice without a causal design
A stakeholder sees that a feature has large SHAP values and asks you to change that feature to improve the outcome.
mediumdocumented fact - Plot ICE curves with the PDP so averages do not hide opposite effects
A partial dependence plot shows a flat or mild effect for a feature that domain experts expect to matter.
mediumdocumented fact - Test whether clusters survive resampling before naming them
A k-means run on customer or route features produces k tidy segments that the business wants to name and target.
mediumdocumented fact - Validate segments on outcomes they were not built from before calling them personas
A clustering of customers or routes is about to become a set of named personas with tailored strategies.
mediumpractitioner heuristic - Derive the false alarm rate an anomaly detector must reach from the base rate and the precision you need
An anomaly detector shows 95% detection with a 1% false alarm rate on a test set, and anomalies occur in 0.1% of cases in production.
mediumdocumented fact - Widen or label scenario forecasts because they exclude uncertainty in the assumed drivers
Finance asks for next quarter's demand under three fuel-price scenarios, each with an 80% interval.
mediumdocumented fact - Guard MASE against zero denominators and state which naive scaling you used
MASE is computed across thousands of SKUs and some values come back infinite or enormous.
mediumdocumented fact - Separate stockout zeros from true zero demand before modelling a retail series
A store-SKU series looks intermittent, and stock records show the item was frequently unavailable.
mediumpractitioner heuristic - Reconcile hierarchical forecasts so route, region and network numbers agree
Route-level and network-level demand forecasts are produced separately, and the routes do not add up to the network total.
mediumdocumented fact - Match the forecasting level of a hierarchy to the noise and the decision
Store-SKU daily sales are too noisy to forecast well, but replenishment decisions are made per store and SKU.
mediumpractitioner heuristic - Check interval coverage by season and segment, not only overall
Overall coverage of a 90% demand interval is 89%, and the team declares it calibrated.
mediumpractitioner heuristic - Pick the forecast quantile from the ratio of under- and over-forecast costs
Operations asks for 'the forecast' to set meal loads per flight, where running out costs more than waste.
mediumdocumented fact - Handle weekly data's non-integer annual season with Fourier terms or STL
A weekly demand series is fitted with a 52-period seasonal model, and the holiday peaks drift by a week every few years.
mediumdocumented fact - Encode moving holidays as windows before and after, not as a single day flag
A Turkish domestic route forecast misses the travel surge around Eid holidays, which move about eleven days earlier each Gregorian year.
mediumdocumented fact - Use bookings already on hand as a known-at-origin input for departure-level forecasts
A revenue-management team forecasts final bookings for a departure 21 days out using only past departures.
mediumempirical finding - Store first-reported and final actuals so forecast accuracy uses the right version
Revenue figures are updated days after the flight date, and accuracy reports are rerun against the latest table.
mediumpractitioner heuristic - Deduplicate late-arriving corrections by business key and load time, explicitly
A flight-day feed sometimes sends the same row twice, the second with an updated revenue figure.
mediumpractitioner heuristic - Minimise personal data in features and give training snapshots a retention period
A churn model's training set copies full customer records, including fields the model never uses, and snapshots are kept indefinitely.
mediumdocumented fact - Report a null result with its interval and detectable effect, not as 'no effect'
A test of a new recommendation model shows no statistically significant revenue change, and the summary says 'the model does not work'.
mediumdocumented fact - Pair any model-driven KPI target with guardrail metrics to resist Goodhart effects
A call-centre routing model is rewarded on average handle time, and handle time falls while repeat calls rise.
mediumpractitioner heuristic - Limit and explain forecast churn between model versions
A new forecasting model version changes next month's plan by 15% for some routes, though inputs barely changed.
mediumpractitioner heuristic - Seed eval sets with questions that invite common misconceptions
A revenue-management help bot is evaluated only on questions written by the people who built it.
mediumempirical finding - Keep your hypothesis out of the prompt when asking an LLM to judge evidence
An analyst asks a model to confirm that a revenue drop on one route was caused by a competitor's new capacity.
mediumempirical finding - Extract from long documents section by section with provenance, then aggregate in code
An analyst asks a model to list every penalty clause from a long supplier contract in one call.
mediumpractitioner heuristic - Enforce validity dates and scope with metadata filters, not with the prompt
The corpus holds several versions of the same policy, for example baggage rules effective before and after 2024-03-01, or tariffs per country.
mediumpractitioner heuristic - Slice retrieval metrics by query type before declaring a retriever better
A hybrid retriever raises average recall@5 by two points over the dense baseline.
mediumpractitioner heuristic - Monitor the ingestion pipeline, because parse failures drop evidence silently
A RAG index is rebuilt nightly from PDFs, HTML and spreadsheets owned by other teams.
mediumpractitioner heuristic - Route counting and aggregation questions to SQL, not to chunk retrieval
Users ask the document assistant "how many flights to FRA were cancelled last month?" or "average fare on IST-ADB in July".
mediumpractitioner heuristic - Budget GenAI features by cost per successful task, not price per token
A business case multiplies expected requests by the list price per million tokens and calls that the cost.
mediumpractitioner heuristic - Check how well the pre-period predicts the metric before counting on CUPED to save traffic
A test plan halves the duration "because we use CUPED", but the metric is first-visit conversion or the population is mostly new customers.
mediumdocumented fact - Count markets, not passengers, when price or capacity is randomised at the route level
A pricing change can only be applied per route, 14 routes are available, and the analyst computes p-values on millions of passengers.
mediumpractitioner heuristic - Analyse only users who could see the change, then dilute the result before forecasting the company-wide effect
A seat-map change reaches 6% of sessions; the all-user test is flat, while a triggered analysis shows +3% on the page.
mediumdocumented fact - Analyse a campaign by assignment first, and read engaged-only numbers with their assumptions attached
An email offer is randomised; only 20% open it, and the analyst compares openers in the treatment arm with everyone in control.
mediumpractitioner heuristic - Keep a long-term holdback to measure what a year of shipped wins really added
A quarterly deck sums the lifts of 14 individually significant experiments and claims +9%, while the dashboard moved +2%.
mediumpractitioner heuristic - Choose the multiplicity family and error rate by what a false positive costs, and say which you used
A route-level analysis tests 80 hypotheses; one analyst uses Bonferroni, another raw p-values, a third Benjamini-Hochberg, and the reports cannot be compared.
mediumdocumented fact - Describe a regression discontinuity effect as local to the cutoff and do not extrapolate it to the whole base
The estimate says a status upgrade raises rebooking by 4 points for members near the threshold, and the sponsor wants to upgrade all 200,000 members.
mediumpractitioner heuristic - Date the treatment from the announcement and check controls for spillover before running a difference-in-differences
A fare change on one route was announced three weeks before it took effect, and a competing route serves as the control.
mediumdocumented fact - Inspect propensity-score overlap and weight extremes before reporting an inverse-probability-weighted effect
An observational analysis of upgrade offers uses inverse-probability weights, and a handful of customers carry weights above 50.
mediumdocumented fact - Quantify how strong an unmeasured confounder must be before dismissing or trusting an observational effect
Customers who use app check-in show a 1.5 times higher ancillary purchase rate, and the room splits between "it works" and "it is just engaged customers".
mediumdocumented fact - Check residual autocorrelation in an interrupted time series and widen the intervals if it is there
A weekly interrupted time series after a schedule change gives a narrow, significant level shift from an ordinary least squares fit.
mediumdocumented fact - Expect a noisy predictor's coefficient to shrink toward zero and measure its reliability before interpreting it
A regression finds that self-reported trips per year barely predicts loyalty spend, and the team concludes that frequency does not matter.
mediumdocumented fact - Report the effective sample size after weighting, because large weight variation shrinks it
A satisfaction survey is weighted to the passenger mix by route and cabin, and the report keeps quoting the raw respondent count in its margin of error.
mediumpractitioner heuristic - Check that a proxy is a cause of the goal and not a consequence of it before making it a target
Hypothetical: call-centre staff are rewarded for short handling time, which correlates with satisfied customers.
mediumdocumented fact - Define outcome metrics on matured cohorts when cancellations, refunds or no-shows arrive late
A booking-flow test reports net bookings after two weeks, but cancellations and refunds keep arriving for up to a month.
mediumpractitioner heuristic - Audit what is logged for each arm when a variant changes the tracking, not only the product
A new checkout variant replaces a page, and its event tracking was rebuilt by a different team.
mediumdocumented fact - Filter on a window function result in an outer query, because WHERE runs before windows
You want only each flight's most recent snapshot and write WHERE ROWNUMBER() OVER (...) = 1 in the same SELECT, or you add a WHERE condition and expect the window to still see all rows.
mediumdocumented fact - Compute a median or percentile over the underlying rows, not as an average of group medians
A dashboard shows median fare per route and a stakeholder asks for the network median, so someone averages the eleven route medians.
mediumdocumented fact - Allocate header-level amounts to the line grain with a written rule before summing by line attributes
An order-like header carries a booking fee or baggage charge once, and a line table has one row per flight segment; the report sums revenue by route from the joined result.
mediumdocumented fact - Use a coverage table and an anti-join to count what did not happen
Someone asks which scheduled flights had no bookings in a week, or which customers received no offer, and the only table you have lists events that occurred.
mediumdocumented fact - Use IS NOT DISTINCT FROM when NULL keys should match in a join or a comparison
You reconcile two extracts by joining on route and fareclass, and rows where fareclass is NULL in both tables vanish from the matched set.
mediumdocumented fact - Do not rely on a UNIQUE constraint to stop duplicates when a key column can be NULL
A table has UNIQUE(customerid, promocode), promocode is nullable, and the nightly load keeps inserting "the same" customer row with no promo.
mediumdocumented fact - Pick the slowly-changing-dimension type per attribute by asking whether history may change past answers
A product team asks for the route dimension to be made "type 2", and every attribute, including a typo-corrected name and a reorganised sales region, starts creating new versions.
mediumdocumented fact - Store validity intervals half-open, and join with start <= t < end
A type 2 dimension stores validfrom and validto as dates, and one version ends on 2024-03-01 while the next begins on 2024-03-01.
mediumpractitioner heuristic - Schedule a snapshot of a mutable source at least as often as it changes, and handle deleted rows explicitly
An operational table holds a booking's current status and is overwritten in place, and a weekly job snapshots it to give "status history".
mediumdocumented fact - Truncate timestamps in an explicit time zone, because date_trunc on a timestamptz uses the session zone
A daily booking count is built with datetrunc('day', createdat), and the same query returns different daily totals when run from a notebook and from a scheduled job.
mediumdocumented fact - Add calendar days and elapsed hours with different interval types
A feature computes "same time next day" with + interval '24 hours', and on the day clocks change the result is off by an hour from the local departure schedule.
mediumdocumented fact - Parameterize each run by its data interval and never read "now" inside a transformation
A daily aggregation filters on CURRENTDATE - 1, and a rerun two days later after a failure silently computes a different day.
mediumdocumented fact - Deduplicate the incoming batch before an upsert, because one statement cannot update a row twice
A load runs INSERT ... ON CONFLICT DO UPDATE from a staging table that contains two versions of the same booking, and the job fails or, in another engine, keeps an arbitrary version.
mediumdocumented fact - Attach an idempotency key to events emitted by retried producers and deduplicate on it downstream
A booking service retries a payment event when a request times out, and some payments are recorded twice in the event table.
mediumdocumented fact - Build hashed surrogate keys from delimited, null-aware fields
A model joins on a hash built by concatenating flightno, date and fareclass, and two different rows produce the same key or a NULL field turns the whole key NULL.
mediumdocumented fact - Break ties explicitly when choosing the latest row per key
A customer table receives two updates in the same second, and the "latest" row chosen with DISTINCT ON or ROWNUMBER ORDER BY updatedat differs between runs.
mediumdocumented fact - Sample by a hash of the entity key so a sample stays consistent across tables and reruns
You need a 5% development sample of bookings and their payments, and sampling each table with its own random draw leaves most bookings without their payments.
mediumpractitioner heuristic - Do not treat LIMIT or TABLESAMPLE SYSTEM as a random sample, and use reservoir sampling for streams
An analyst profiles a billion-row table with SELECT LIMIT 100000 or TABLESAMPLE SYSTEM (1) and reports distributions from it.
mediumdocumented fact - Check downstream consumers through lineage before renaming or retyping a column
A source team renames fareamount to basefare in the booking table and says "only the mart uses it".
mediumpractitioner heuristic - Set data-test severity by blast radius and keep the failing rows
A pipeline has forty dbt-style tests that all fail the build, and a harmless null in an optional column blocks the daily revenue refresh.
mediumdocumented fact - Record run-level lineage so an incident can be traced to the job run that produced a bad table
A revenue table shows a spike, and the team cannot tell which of three overnight jobs, running against which input versions, wrote it.
mediumdocumented fact - Replace a high-cardinality code with its descriptive attributes so new codes inherit meaning
A model uses airport, aircraft-type or merchant codes with hundreds of levels, and new codes arrive with no history.
mediumpractitioner heuristic - Count populated cells before adding a crossed categorical feature
You cross two or three categoricals, such as route by weekday by fare class, to give a linear model an interaction or to build a composite key.
mediumpractitioner heuristic - Replace two near-duplicate prices with one level and one gap before interpreting a linear model
Own fare and competitor fare move together, both sit in a linear or logistic model, and their coefficients flip sign between retrains.
mediumpractitioner heuristic - Build calendar features for each timestamp that drives the outcome, such as booking date and travel date
A booking table has both booking date and departure date, but only departure-date calendar features exist because it is the row key.
mediumpractitioner heuristic