Statistics, experimentation, causal reasoning · insight-0034
State the base rate before quoting a detector's accuracy to stakeholders
- Evidence type
- empirical finding
- Difficulty
- easy
- Roles
- Data scientist, Data analyst, ML engineer
- Industries
- banking, aviation, healthcare
Summary: situation, action, reason, boundary
Situation
A fraud or no-show alert is described as "95% accurate", and a manager plans staffing around its flags.
Action
Present expected outcomes per 10,000 cases in natural frequencies: how many true cases, how many flagged, how many flags are correct. Choose the threshold from those counts and the cost of each error.
Reason
Precision depends on prevalence as well as sensitivity and specificity; natural frequencies make that dependence visible to non-specialists.
Boundary
Fraud, anomaly alerts, medical screening, churn warnings and rare-event classifiers.
Symptom or risk
At a low base rate, most flags are false positives, so staff are overwhelmed and trust in the system collapses.
Exceptions
Balanced problems where positives are common need less emphasis on base rates.
Trade-offs
Concrete counts depend on an assumed prevalence that may shift; show a range.
Pitfalls
Using the training-set prevalence after resampling instead of the deployment prevalence.
Example
Hypothetical: prevalence 1%, sensitivity 90%, specificity 95%: of 10,000 bookings, 90 frauds are caught and about 495 legitimate bookings are flagged, so roughly 15% of flags are correct.
Lessons and concepts
Claims and their sources
Presenting probabilistic information as natural frequencies improves people's Bayesian reasoning compared with single-event probabilities.