Classical ML, validation, metrics, interpretation, debugging · insight-0194
Derive the false alarm rate an anomaly detector must reach from the base rate and the precision you need
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer, Data analyst
- Industries
- banking, manufacturing, telecom
Summary: situation, action, reason, boundary
Situation
An anomaly detector shows 95% detection with a 1% false alarm rate on a test set, and anomalies occur in 0.1% of cases in production.
Action
Work backwards: from prevalence , detection rate and target precision , the maximum false alarm rate is . If the detector cannot reach it, add a second-stage filter (rules or a supervised model) rather than more reviewers.
Reason
When anomalies are rare, even a small false alarm rate applied to the huge normal population swamps the true detections.
Boundary
Fraud, intrusion, equipment fault, data-quality and transaction monitoring.
Symptom or risk
About ten false alarms per true anomaly reach analysts; they learn to ignore alerts and real events are missed.
Exceptions
If missing an event is catastrophic and reviews are cheap, a low-precision detector can still be worth it; make that trade explicit.
Trade-offs
Lower false alarm rates usually cost detection; the cost matrix should decide.
Pitfalls
Measuring precision on a test set enriched with anomalies, which overstates production precision.
Example
Hypothetical: at 0.1% prevalence and 95% detection, 1% false alarms give precision near 9%; reaching 50% precision needs a false alarm rate below about 0.1%.
Lessons and concepts
Claims and their sources
Because of the base-rate fallacy, the false alarm rate limits detector effectiveness; a useful P(intrusion | alarm) requires a very low false alarm rate.