Statistics, experimentation, causal reasoning · insight-0004
Run experiments in whole weekly cycles so the weekday mix matches across the decision
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Data analyst
- Industries
- aviation, retail, e-commerce
Summary: situation, action, reason, boundary
Situation
An experiment reaches its sample size on a Tuesday after starting on a Friday, and the team wants to close it.
Action
Plan the duration as a whole number of weeks that also meets the sample size. If behaviour has monthly or payday cycles, check the effect by week before generalizing.
Reason
Treatment effects often differ by day type. A partial week weights some day types more than the population the launch will serve, even if randomization is perfect.
Boundary
Consumer products, booking funnels, retail and any traffic with strong day-of-week patterns.
Symptom or risk
The sample over-represents weekend leisure travellers or weekday business users, so the estimated effect describes a mix the business never sees for a full week.
Exceptions
Internal tools with flat weekday traffic, or tests whose sample size is reached in hours and whose population has no weekly cycle.
Trade-offs
Rounding up to full weeks costs a few days of traffic in exchange for an estimate that matches the deployment mix.
Pitfalls
Rounding up to a full week only after seeing a non-significant result, which turns the rule into peeking.
Example
Hypothetical: if Friday and Sunday sessions are mostly leisure shoppers, a Friday-to-Tuesday test counts two of them and only two business weekdays.