S09 · defense simulation

Defend a delivery-fee experiment before it is trusted

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

A food-delivery marketplace (simulated) tests a lower delivery fee. You wrote the experiment design and the readout; a panel of the head of product, a senior statistician and the marketplace-operations lead will challenge it. All numbers are simulated.

Your design note says:

Unit: each eligible customer is randomised once, on first session in the test, 50/50.
Metric: ordered within 7 days of first session. Baseline 4.0%; smallest effect of interest +0.4 points (4.4%).
Error rates: 5% two-sided, 80% power. Eligible customers arrive at about 6,000 a day.
Duration: 14 days of assignment, then a 7-day outcome window.
Couriers are shared by both arms.

Stage 1 of 4

The head of product: "Why 14 days and why 40,000 a head? Finance says we can only afford 20,000 per arm. Is that enough?"

How do you defend the size and duration?

For 4.0% against 4.4% at 5% and 80% power, about 39,500 per arm are needed; at 3,000 per arm a day that is 13.2 days, rounded to 14.Fourteen days is the usual length for an online experiment, and two weeks is long enough to cover a weekly cycle, so the plan follows common practice.20,000 per arm is enough: it is what we can afford, and a 0.4-point lift on a 4% base will still show up in the data if it is real.