Statistics, experimentation, causal reasoning · insight-1024
Quantify how strong an unmeasured confounder must be before dismissing or trusting an observational effect
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- healthcare, aviation, retail
Summary: situation, action, reason, boundary
Situation
Customers who use app check-in show a 1.5 times higher ancillary purchase rate, and the room splits between "it works" and "it is just engaged customers".
Action
Report the E-value for the point estimate and for the confidence limit closest to no effect. Compare it with the association strength of the strongest measured confounders, and state whether a plausible hidden factor could reach that level.
Reason
The E-value is the minimum strength of association an unmeasured confounder would need with both treatment and outcome to explain the result away, which turns a vague worry into a number to argue about.
Boundary
Observational risk-ratio style estimates where randomisation is not available.
Symptom or risk
The argument is unfalsifiable on both sides and the decision follows the louder voice.
Exceptions
Estimates on scales that cannot be converted to approximate risk ratios without assumptions.
Trade-offs
It bounds one kind of bias and says nothing about selection or measurement error.
Pitfalls
Reading a large E-value as proof of causation.
Example
A risk ratio of 1.5 gives an E-value of 1.5 + sqrt(1.5 x 0.5), about 2.37.
Lessons and concepts
Claims and their sources
The E-value is the minimum strength of association, on the risk-ratio scale, that an unmeasured confounder would need with both treatment and outcome to fully explain away an association given measured covariates, and it should be computed for the estimate and the confidence limit closest to the null.