Statistics, experimentation, causal reasoning · insight-1028
Do not treat a very large opt-in sample as more trustworthy than a small random one
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, Data analyst
- Industries
- aviation, retail, public-sector
Summary: situation, action, reason, boundary
Situation
A post-trip rating prompt collects 60,000 voluntary responses out of 2 million passengers, and the average is presented with a tiny confidence interval.
Action
Compare respondents with the passenger base on known attributes, estimate how strongly response relates to the outcome with a small random follow-up sample, and prefer a modest probability sample for headline levels. Use the large opt-in data for trends and relative comparisons with that caveat.
Reason
Error of a sample average splits into data quality, the correlation between the outcome and the response indicator, data quantity and problem difficulty; with non-random response the error relative to the usual benchmark grows with population size.
Boundary
Voluntary surveys, app ratings, support-contact analysis and opt-in panels.
Symptom or risk
Whether people respond is related to how they felt, so the average is biased, and more responses make the interval narrower around the wrong value.
Exceptions
Response mechanisms plausibly unrelated to the outcome, which is hard to demonstrate.
Trade-offs
Probability samples cost money and have their own non-response.
Pitfalls
Quoting a margin of error computed as if responses were a random draw.
Example
Hypothetical: with a response-outcome correlation of 0.05, 60,000 of 2,000,000 respondents give an error near 0.28 standard deviations, versus about 0.03 for a random sample of 1,500.
Lessons and concepts
Claims and their sources
The error of a sample average decomposes into data quality (the correlation between the outcome and the recording or response indicator), data quantity and problem difficulty, and without probability sampling the error relative to the 1/sqrt(n) benchmark increases with the population size.