Statistics, experimentation, causal reasoning · insight-0012
Report the numerator and denominator next to every ratio metric in an experiment
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- e-commerce, aviation, saas
Summary: situation, action, reason, boundary
Situation
Click-through rate or revenue per search improves in treatment, and nobody has looked at clicks and searches separately.
Action
Show the ratio, its numerator and its denominator as separate count metrics, each with an interval. Interpret the ratio only after confirming the denominator did not move, or explain both movements together.
Reason
A ratio compares like with like only if the treatment leaves the denominator population unchanged.
Boundary
CTR, conversion per session, revenue per booking, load factor and any per-unit KPI.
Symptom or risk
The ratio rose because the denominator fell, for example fewer searches after a layout change, so a worse product looks better.
Exceptions
When the denominator is the randomization unit itself (per assigned user), it is controlled by design.
Trade-offs
More numbers on the scorecard; fewer misread wins.
Pitfalls
Mixing ratio-of-sums and average-of-ratios definitions between dashboard and experiment, which can disagree in sign.
Example
Hypothetical: a fare-filter change cuts low-intent searches, so bookings per search rise while total bookings stay flat.
Lessons and concepts
Claims and their sources
Ratio metrics can be misleading or invalid when the denominator changes; defining count metrics for the numerator and denominator and reporting them alongside the ratio detects this.