Experimentation and causality

Interview question

midlevel
8 minsuggested time
2Data scientist, Data analyst

A team ran an A/B test planned for 14 days but checked the p-value every day and stopped on day 6 when it dropped below 0.05. They want to ship. What is the problem, and what would you recommend for future tests?

Tests: Optional stopping (peeking) and its effect on false-positive rates; sequential designs; practical experiment governance.

day 4 day 5