Skip to content
ML Academy

project-001 · Analytics and statistics · aviation

Booking-flow A/B test: should the one-page passenger form ship?

The digital team of the fictional carrier ZX replaced the two-step passenger-details form with a single page. A randomized test ran for 14 days. Should the new form ship to all traffic, given its effect on completed bookings per visitor and on two guardrails (payment errors per session and revenue per visitor)?

Dataset
Synthetic booking-funnel experiment (generated by reference r01; seeded, known true effect)
Licence
synthetic - generated by academy code; no third-party data
Reference run
executed

At a glance

  • Data

    Synthetic booking-funnel experiment (generated by reference r01; seeded, known true effect) (synthetic) · Default about 200,000 visitors (one row each) plus a session table; configurable; fits in memory on a laptop

    Details on data

  • Baseline

    The decision baseline is not shipping (status quo). The statistical baseline is the plain difference in conversion proportions between arms with an unpooled standard…

    Details on baseline

  • Validation

    This is estimation, not prediction, so validation means checking the estimator against known truth: Repeat the generator over at least 500 seeds per scenario: CI…

    Details on validation

  • Metric

    Absolute lift (pp) and relative lift with 95% CI; SRM p-value; achieved power; CUPED variance reduction (%);

    Details on metric

Success criterion

A decision memo that meets all of these, written before unblinding where marked:

  1. Pre-registered design (before unblinding): primary metric, unit, α=0.05\alpha = 0.05 two-sided, 80% power, and a minimum practically important lift of +0.3 percentage points on a 4.0% control conversion (both numbers hypothetical), with the sample size computed from them.
  2. Trust gate: a sample-ratio-mismatch (SRM) chi-square test on assignment counts; if p<0.001p < 0.001 the memo stops and diagnoses instead of reporting a lift.
  3. Estimate and rule: absolute and relative lift with 95% CI, and the ship / do-not-ship / extend rule applied to where the CI sits relative to 0 and to +0.3 pp.
  4. Estimator check on the simulator: across at least 500 seeded replications, the 95% CI covers the known true effect in 93-97% of runs, and the A/A (true effect = 0) false-positive rate is 3-7%.

Context

All data are synthetic: a seeded generator in the reference implementation simulates visitors to ZX's booking flow with a known true effect, so the analysis can be graded against ground truth (real experiments never reveal it). Randomization is per visitor; visitors may have several sessions. Booking conversion is the upstream lever of revenue management: a booking that is later cancelled still counts here, which is a known gap between this metric and flown revenue.

The generator has switchable scenarios that reproduce classic failures: a bot filter applied to one arm only (SRM), a novelty effect that decays over the first days, and a null scenario for A/A checks. Traffic volume, fares and the practical threshold are hypothetical.

Data

Dataset
Synthetic booking-funnel experiment (generated by reference r01; seeded, known true effect) (synthetic)
Licence
synthetic - generated by academy code; no third-party data
Size
Default about 200,000 visitors (one row each) plus a session table; configurable; fits in memory on a laptop
Verification
unverified

Access

Generated locally by python/reference/r01_ab_test_analysis (specified, not yet implemented). No download, no account. Scenario and seed are set in the module's config so every run is reproducible. Fallback: none needed; if the reference is not yet implemented, the learner writes the generator from the dictionary below.

Grain

visitors: one row per randomized visitor for the 14-day window (the analysis unit equals the randomization unit). sessions: one row per session, used only for ratio guardrails whose denominator is sessions.

ColumnMeaning
visitor_idRandomization unit; stable pseudonymous id
variantcontrol or treatment, assigned 50/50 by hashing visitor_id
first_exposure_dateDate the visitor first saw the passenger form (trigger point)
devicemobile or desktop at first exposure (pre-treatment)
marketdomestic, intl-short or intl-long, from the searched route (pre-treatment)
pre_period_bookingsBookings in the 28 days before the test; covariate for CUPED
sessionsSessions during the test (post-treatment; never a covariate)
booked1 if the visitor completed at least one booking during the test (primary metric)
revenue_eurBooking revenue in EUR, hypothetical fares; heavy-tailed
payment_errors / payment_attemptsSession-level guardrail counts (ratio metric)

EDA and data-quality rules

  • Assignment integrity: every visitor_id has exactly one variant, and exposure dates fall inside the test window.

  • SRM gate: chi-square goodness-of-fit of visitor counts against the 50/50 design before any outcome is looked at; p < 0.001 stops the analysis.

  • Filters (bots, internal traffic, test accounts) are defined before unblinding and applied identically to both arms; report counts removed per arm.

  • Randomization check on pre-treatment variables (pre_period_bookings, device, market): standardized differences, not a search for significant p-values.

  • Revenue outlier rule (winsorize at a pre-registered percentile such as 99.9) is fixed before unblinding and applied to both arms.

  • Daily treatment-minus-control plot to detect novelty or primacy effects and logging breaks.

Baseline

The decision baseline is not shipping (status quo). The statistical baseline is the plain difference in conversion proportions between arms with an unpooled standard error and a 95% Wald or Newcombe interval. Every refinement (variance reduction, bootstrap) must agree with or explain its difference from this number.

Features

No predictive features. Analysis variables are defined in the pre-registration:

  • Primary: booked per visitor (binary). Secondary: bookings per visitor; winsorized revenue per visitor.
  • Guardrails: payment errors per payment attempt (a ratio of two per-visitor sums, so its variance needs the delta method or a visitor-level bootstrap), and revenue per visitor (non-inferiority margin, hypothetical).
  • Covariate: pre_period_bookings (pre-treatment) for CUPED. Post-treatment variables such as sessions or "reached the payment page" are never covariates or filters.
  • Pre-registered segments: device and market. Anything else is exploratory.

Candidate methods

  • Difference in proportions with a Newcombe (Wilson-based) interval

    Matches the pre-registered design, needs no model, and is the number stakeholders can check by hand.

  • CUPED-adjusted difference using pre-period bookings

    Remains unbiased under randomization and narrows the interval when the covariate correlates with the outcome; worth it only if the variance reduction is material, which the simulator quantifies.

  • Visitor-level bootstrap for revenue and ratio guardrails

    Revenue is heavy-tailed and ratio metrics are not visitor-level means; the bootstrap is a check on the delta-method intervals rather than a replacement.

Validation design

This is estimation, not prediction, so validation means checking the estimator against known truth:

  • Repeat the generator over at least 500 seeds per scenario: CI coverage of the true effect, A/A false-positive rate, and empirical power at the pre-registered effect size.
  • Peeking study: apply a naive daily significance check to the same replications and report how much it inflates the false-positive rate compared with a single analysis at the planned end.
  • Scenario tests: the SRM gate must trigger in the bot-filter scenario and stay silent in the clean one.

Leakage checks

Conditioning on post-treatment behaviour is the leakage analogue here: filtering to "visitors who reached the form" or adjusting for sessions changes who is compared. A triggered analysis is valid only if the trigger is logged identically in both arms (the control arm logs a counterfactual trigger).

Bias and representativeness

SRM from asymmetric filtering, novelty effects, cross-device duplicates (one person, two visitor ids, possibly both arms), and segment fishing. Segment results beyond the two pre-registered ones are labelled exploratory.

Robustness checks

Re-estimate with a different winsorization percentile, excluding the first two days, per week, and with a permutation test; the decision must not flip under these variations, or the memo says it is fragile.

Technical metrics

Absolute lift (pp) and relative lift with 95% CI; SRM p-value; achieved power; CUPED variance reduction (%); simulated coverage, A/A false-positive rate and inflation from peeking.

Business interpretation

Incremental bookings per 100,000 visitors and incremental weekly revenue under hypothetical traffic and fares, each shown at the CI bounds, not only the point estimate; guardrail deltas against their non-inferiority margins.

Error slices

  • Device (pre-registered): a mobile-only gain can hide a desktop loss.

  • Market (pre-registered): intl-long visitors book less often, so absolute lifts differ even with a common relative effect.

  • Exposure day 1-2 versus later days (novelty scenario).

  • Bot-filter scenario: compare the arm-level counts removed by each filter to locate the SRM source.

Implementation steps

  • Write config.yaml as the pre-registration: metrics, unit, alpha, power, practical threshold, guardrail margins, segments, stopping rule; compute sample size with statsmodels power functions.

  • Generate the data with a fixed seed and scenario flag (clean, srm, novelty, null); store the true effect separately from the analysis tables.

  • Run integrity checks: assignment uniqueness, SRM test, pre-treatment balance; stop with a diagnostic report if the SRM gate fails.

  • Estimate the primary effect (difference in proportions), then CUPED; estimate guardrails with the delta method and confirm with a bootstrap.

  • Apply the pre-registered decision rule and write the memo: estimate, interval, rule outcome, guardrails, one interpretation limit.

  • Run the simulation study (coverage, A/A, power, peeking) and save its table next to the memo.

Reproduction

Reference (specified, not yet implemented): cd python && uv run python -m reference.r01_ab_test_analysis.run. Planned outputs: a memo, a metrics JSON and the simulation table in the module's output folder. Seeds and scenario live in the config; dependencies are pinned by python/uv.lock. Target runtime: under 2 minutes on a laptop CPU, including the 500-replication study.

Limitations

  • Synthetic data with a known, simple data-generating process; real experiments add identity stitching, interference and logging bugs that the simulator only partly imitates.

  • Conversion counts bookings, not flown revenue; later cancellations and fare mix are outside the metric.

  • The practical threshold, fares and traffic are hypothetical; the economics illustrate the method, not ZX's business.

  • A 14-day test cannot measure long-term effects such as repeat booking.

Production and monitoring

In a real platform: assignment and exposure logging owned by the experimentation service, an automated SRM alert on day 1, guardrail dashboards, a ramp plan (for example 5% -> 50% -> 100%), a small long-running holdback to check that the effect persists, and a decision log entry that records the pre-registration, result and decision.

Interview questions

  • Your lift is +0.25 pp with p = 0.03, but the SRM test gives p = 0.0004. What do you tell the product manager, and what do you investigate first?

  • You randomize by visitor but the guardrail is errors per session. Why is the naive standard error wrong, and how do you fix it?

  • The PM wants to stop on day 4 because the result is significant. Explain the risk with numbers from a peeking simulation.

  • When does CUPED narrow the interval, and can it ever bias the estimate?

Postmortem and alternatives

The most common real failure is discovering an SRM after the result has been shared; putting the gate before unblinding prevents that. If traffic cannot reach the planned sample, the alternative is a sequential design with always-valid inference or a longer test, not a lower practical threshold. A surrogate metric (form completion) would reach significance faster but needs evidence that it moves bookings.

Reference implementation

Status: executed — outputs below are from the recorded run

cd python && uv run python -m reference.r01_ab_test_analysis.run

r01 results - A/B test analysis of a booking-flow change (synthetic data, known true effect)

Recorded run of cd python && uv run python -m reference.r01_ab_test_analysis.run (full mode, 200,000 visitors per experiment, 500 replications per scenario). Every number below is copied from outputs/metrics.json, outputs/memo.md, outputs/daily_effect_*.csv and outputs/simulation_study.csv, written by that run. The data are synthetic (seeded generator with a known true effect); traffic, fares, the +0.3 pp practical threshold and the guardrail margins are hypothetical teaching values for the fictional carrier ZX.

ItemValue
Run date2026-10-11
Seedmaster seed 20261011 (one SeedSequence; every scenario and replication owns a child, so the worker count does not change results)
Python / numpy / pandas / scipy / statsmodels3.12.13 / 2.5.3 / 3.0.6 / 1.18.1 / 0.15.0
Runtime of the recorded run44.6 s wall (46.8 s by time, 173.9 s CPU) with 8 worker processes on a 10-core laptop that was shared with other jobs; the five 500-replication studies took 10.7 / 9.0 / 7.4 / 6.8 / 6.8 s
Scenariosclean (true +0.5 pp), srm (bot filter logged in the treatment arm only), novelty (early lift decays to +0.5 pp), aa_null (true effect 0), mde (true effect equal to the +0.3 pp threshold)

Findings

  1. Design. Sample size for +0.3 pp on a 4.0% control rate, alpha 0.05 two-sided, 80% power: 69,376 visitors per arm by the closed form and 69,358 from statsmodels (Cohen's h = 0.01504). After filtering, the clean experiment has 93,429 control and 93,620 treatment visitors, which gives 90.2% power at exactly +0.3 pp.

  2. The trust gate passes in the clean scenario and is checked before any outcome is read. All five integrity checks hold (unique ids, valid variants, exposure dates in the window, assignment equals the hash rule, session table sums match visitor totals). SRM chi-square on raw assignment: p = 0.436; on the analysis population (after the bot and internal-traffic filters, which removed 5,873 / 561 from control and 5,996 / 616 from treatment): p = 0.659, treatment share 0.5005. Largest standardized pre-treatment difference 0.0070.

  3. Estimate and rule (clean). Control 3.929%, treatment 4.516%: absolute lift +0.587 pp (95% Newcombe CI +0.405 to +0.769 pp), relative lift +14.9% (+10.3% to +19.6%). The unpooled Wald interval agrees to three decimals (+0.405 to +0.769 pp). The lower bound is above the +0.3 pp threshold, so the pre-registered rule says ship. The known true effect is +0.500 pp; the estimate is 0.087 pp too high (one standard error is 0.093 pp) and the interval covers the truth. The permutation p-value sits at its floor (0.001, 1,000 draws).

  4. CUPED is not worth using on this generator. Using pre_period_bookings, the CUPED lift is +0.573 pp (+0.392 to +0.754 pp), a variance reduction of 1.55% in the single experiment and 1.53% on average over the 500 clean replications (mean interval width 0.366 pp raw, 0.363 pp CUPED). The pre-period covariate carries little information about a 4% conversion; this is a property of the simulation, not a benchmark.

  5. The payment-error guardrail is not demonstrated, so "ship" comes with a condition. Errors per payment attempt: control 2.81%, treatment 3.18%, difference +0.37 pp, 95% delta-method CI -0.16 to +0.89 pp against a +0.5 pp margin: inconclusive. The visitor-level bootstrap gives -0.16 to +0.89 pp, the same interval. The delta-method SE is 1.01 times the naive SE, so clustering of attempts within visitors barely matters in this generator. With 8,032 control and 8,615 treatment attempts, about 110,123 attempts per arm would be needed to demonstrate non-inferiority if the true increase were the hypothetical +0.3 pp. Winsorized revenue per visitor (cap at the 99.9th percentile, EUR 1,030.78): +1.447 EUR (CI +0.903 to +1.991; margin -0.155 EUR): non-inferiority demonstrated. The memo's recommendation is therefore a staged ramp with an alert on payment errors, not an unconditional ship.

  6. Revenue per visitor depends on the winsorization percentile. Difference at p99 +0.815 EUR, p99.9 +1.447 EUR, p99.99 +1.655 EUR, no winsorization +1.799 EUR. The sign and the guardrail verdict do not change, but the magnitude moves by a factor of two, which is the usual heavy-tail problem; the memo should quote the pre-registered p99.9 value and the range.

  7. Segments (pre-registered: device, market). Mobile +0.72 pp (+0.50 to +0.93), desktop +0.38 pp (+0.06 to +0.70); domestic +0.68 pp, intl-short +0.48 pp, intl-long +0.40 pp. The device interaction (mobile minus desktop) is +0.34 pp with p = 0.090, so it is not significant at this sample size even though the generator builds in a stronger effect on mobile (relative-lift multipliers 1.5 vs 0.3). Failing to detect heterogeneity is not evidence of its absence.

  8. Robustness (clean). All six variants give the same decision (ship): Newcombe, Wald, CUPED, excluding the first two exposure days (+0.592 pp), week 1 only (+0.614 pp, CI +0.354 to +0.874) and week 2 only (+0.559 pp, CI +0.304 to +0.815). Week 2 minus week 1 is -0.054 pp (p = 0.77). Single exposure days are noisy: six of the 14 daily intervals exclude zero.

  9. Single experiments per scenario (one draw each; the last column is the known truth, not available to the analyst):

    ScenarioSRM p (analysis population)Lift, 95% CI (pp)Rule outcomeTrue 14-day effect (pp)
    clean0.659+0.587 (+0.405, +0.769)ship+0.500
    srm7.9e-49 (gate failed)not reportedstop and diagnose+0.500
    novelty0.127+0.830 (+0.646, +1.015)ship+0.711 (steady state +0.500)
    aa_null1.000-0.043 (-0.221, +0.135)do not ship0.000
    mde0.866+0.290 (+0.111, +0.470)extend+0.300

  10. The SRM gate catches the one-arm bot filter and there is no valid lift to report. Raw assignment is balanced (p = 0.214, 100,278 vs 99,722) but the filters removed 613 visitors from control and 6,543 from treatment (5,952 bots, all from treatment), leaving 99,665 vs 93,215 (p = 7.9e-49). If the gate were bypassed, the number would be +0.84 pp (+0.66 to +1.02) and the rule would say ship, 0.34 pp above the truth of +0.50 pp; over 500 replications that ungated interval covers the truth 27% of the time and says ship in 99.8% of replications.

  11. The novelty scenario shows what a 14-day window measures. The 14-day estimate is +0.83 pp, the day-0 lift is +2.06 pp (CI +1.33 to +2.80), the 14-day truth is +0.71 pp and the steady-state effect shipping would deliver is +0.50 pp. The interval covers the 14-day truth but not the steady state; the analysis cannot distinguish them without a longer test or a holdback.

  12. Simulation study (500 replications per scenario), the record's estimator check.

    ScenarioTrue effect (pp)Mean estimate (pp)CI coverage (Newcombe)Significant rateSRM triggeredDecision shares
    clean+0.500+0.50296.2%100%0.0%ship 58.8%, extend 41.2%
    srm+0.500+0.73527.0%100%100.0%ship 99.8%, extend 0.2% (ungated)
    novelty+0.712+0.71595.6%100%0.0%ship 99.4%, extend 0.6%
    aa_null0.000+0.00193.4%6.6%0.6%do not ship 90.6%, extend 9.4%
    mde+0.300+0.30093.8%88.6%0.2%extend 93.8%, do not ship 3.2%, ship 3.0%

    Monte-Carlo standard errors are 0.9 to 1.1 percentage points for these proportions (2.0 for the 27% coverage). Against the record's criteria: coverage in the clean scenario 96.2% (inside 93-97%); A/A false-positive rate 6.6% (inside 3-7%, close to the upper edge and consistent with 5% at this number of replications); SRM triggers in 100% of the SRM scenario and in 0% of the clean scenario. Empirical power at the practical threshold is 88.6%, against 90.2% from the formula for the same sample size.

  13. Peeking inflates false positives and exaggerates the effect. Applying a naive daily significance check (pooled z test on cumulative visitors, 14 looks) to the A/A replications flags a "significant" result in 21.0% of runs against 6.6% for a single analysis at the planned end; a Bonferroni-corrected check flags 2.2%. In the clean scenario the estimate at the first significant look averages +0.623 pp against a truth of +0.500 pp, with a median stop at look 2. With the true effect at the threshold (mde), naive peeking finds significance in 92.0% of runs against 88.6% at the end.

  14. A "ship" in one experiment does not mean the rule ships reliably. With a true effect of +0.5 pp, the rule (lower bound at least +0.3 pp) says ship in 58.8% of replications and extend in 41.2%. The sample size was sized to detect +0.3 pp against zero, not to show a lower bound of +0.3 pp when the truth is +0.5 pp. The single clean experiment above happened to land in the ship group.

What this does not prove

  • It does not show that the one-page form helps real visitors. The effect is built into a generator with independent visitors, one conversion probability per visitor and a multiplicative lift; real tests add identity stitching (one person, two visitor ids, possibly in both arms), interference, logging bugs and drifting traffic that this generator imitates only through the bot filter and novelty scenarios. Cross-device duplicates are not simulated.
  • Coverage of 96.2% and a false-positive rate of 6.6% describe these estimators on this generator at this traffic level. They do not certify the same intervals on skewed or clustered real metrics. Five hundred replications cannot separate a 5% error rate from 6.6%.
  • The payment-error margin is not demonstrated, and at about 8,000 attempts per arm it cannot be; this is a power limit of the test, not evidence of harm (the true increase in the generator is +0.3 pp, inside the margin).
  • CUPED's 1.5% variance reduction is a property of how weakly pre_period_bookings predicts booking here. It says nothing about covariates that are more informative.
  • The peeking study analyses visitors by first-exposure day with their final outcomes, a simplification of daily data pulls, and it demonstrates the problem without implementing a sequential design (always-valid inference is only described in the project record).
  • Conversion counts bookings, not flown revenue; later cancellations, fare mix and repeat booking are outside a 14-day window. The EUR figures are hypothetical business translations, not a statement about any carrier.
  • The single-experiment rows are one draw each; the study table, not those rows, is the evidence about the methods.
  • The segment results are the two pre-registered cuts only; no other slicing was done, so none of them is a fishing result, but each segment interval is wider than the overall interval.

Reproduce

cd python && uv run python -m reference.r01_ab_test_analysis.run
cd python && uv run pytest tests/test_r01_ab_test_analysis.py

Builder-authored reference. It is not learner evidence and is never credited to your portfolio.

Rubric4 criteria

CriterionMeetsExceeds
Pre-registrationMetric, unit, alpha, power, practical threshold and sample size are fixed before outcomes are inspected.Also fixes guardrail margins, segments and filters, and explains why the threshold is the business-relevant one.
Trustworthiness checksRuns the SRM test and a pre-treatment balance check before estimating.Locates the SRM source in the bot scenario and shows the estimate is unusable rather than 'adjusting' it.
Estimation and decisionReports absolute and relative lift with CI and applies a stated rule that uses the practical threshold.Validates coverage, A/A error and peeking inflation by simulation and handles the ratio guardrail correctly.
CommunicationOne-page memo with the decision, interval and one limitation.States the business impact at both CI bounds and what would change the decision.

References7