Classical ML, validation, metrics, interpretation, debugging · insight-0137

Benchmark against the incumbent business rule, not only against a trivial baseline

Situation
A model beats a dummy baseline and you are drafting the launch proposal.
Action
Encode the current rule (for example, overbook by last year's no-show rate per route) and evaluate it on the same holdout, metric and cost units as the model.
Reason
The decision is to replace the incumbent, so the relevant lift is model minus incumbent, measured in business terms.
Boundary
Replacing manual rules, spreadsheets or vendor scores in pricing, risk, forecasting and triage.

Symptom or risk

The existing analyst rule or heuristic already captures most of the value; the model adds cost, latency and maintenance for little gain.

Where no process exists yet, a simple transparent rule you design is the comparison point.

A small lift can still justify ML at scale; a large lift can still fail on explainability or operational fit.

Evaluating the incumbent on data it influenced (its own decisions changed outcomes) without acknowledging the bias.

Google's Rules of ML recommend not being afraid to launch without ML, starting with heuristics and keeping the first model simple.
Rules of Machine Learning: Best Practices for ML Engineering