Statistics, experimentation, causal reasoning · insight-1032

Audit what is logged for each arm when a variant changes the tracking, not only the product

Situation
A new checkout variant replaces a page, and its event tracking was rebuilt by a different team.
Action
Check sample ratio and the counts of every key event per arm before reading outcomes, compare client and server counts for the same actions, and keep a metric computed from a source independent of the changed code.
Reason
Sample ratio mismatches are symptoms of many data-quality problems, and telemetry that a variant adds, removes or delays changes which users produce events at all.
Boundary
UI rebuilds, new clients and any test that touches instrumentation.

Symptom or risk

The variant appears to lose or gain conversions because events are dropped, double-fired or delayed, and the difference is a measurement artefact.

Server-side outcomes recorded outside the variant's code path.

Parallel logging costs engineering time.

Comparing client-side events across arms when only one arm uses a new client library.

A sample ratio mismatch is a symptom of many data-quality issues and ignoring it can make a bad change look good; the paper offers a taxonomy of causes built from experiments at four companies.
Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners