Engineering, production, governance, cost, communication · insight-1299
Walk stakeholders through twenty concrete predictions before debating the metric
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Data analyst, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A review meeting keeps stalling on whether an F1 of 0.71 is good (hypothetical).
Action
Prepare about twenty real cases drawn without cherry-picking: random ones plus the costliest errors. Ask stakeholders what they would do with each prediction, record every disagreement as a requirement, and turn those into testable acceptance rules.
Reason
People judge usefulness by consequences of concrete cases, and doing so surfaces acceptance criteria that no aggregate metric states.
Boundary
Early and pre-launch reviews with business owners and operations teams.
Symptom or risk
The metric hides a requirement mismatch; stakeholders learn at launch that the errors they care about are the ones the model makes.
Exceptions
Where cases are sensitive, use masked or synthetic examples.
Trade-offs
Takes an hour of senior time and covers few cases, so it complements rather than replaces metrics.
Pitfalls
Showing only the model's best cases.