S10 · stats trap simulation
The checkout test that was significant on day 6
People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.
- 5stages
- 35 minabout
- 0stages committed
The brief
An online retailer (simulated) tests a redesigned checkout page against the current one. The plan, written before launch: 14 days, 5,000 visitors per arm per day (70,000 per arm), primary metric order conversion, two-sided test at the 5% level; 70,000 per arm gives about 80% power for a 0.3-point absolute lift on a 3.9% base rate. The dashboard shows cumulative results each day, and the team has been checking it daily. All counts are simulated.
Stage 1 of 5
On day 6 a product manager (simulated) posts in chat: "Redesign is significant: 4.23% against 3.90%, p = 0.039. Can we ship it today?" The cumulative counts on day 6 were 1,270 orders of 30,000 visitors (redesign) against 1,170 of 30,000 (current).
What do you ask first?