Engineering, production, governance, cost, communication · insight-1281
Agree the fairness criterion with business and risk owners before auditing groups
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, Applied AI
- Industries
- banking, insurance, public-sector
Summary: situation, action, reason, boundary
Situation
A default-risk model is audited by applicant group, and the report lists calibration, false-positive rate and approval rate for two groups whose default prevalence differs (hypothetical).
Action
Before computing anything, name the harm that matters (wrongly denied, wrongly approved, unequal approval), choose the primary criterion and tolerated gap with the decision owner and legal, and report the others as context rather than targets.
Reason
When outcome prevalence differs between groups, a score that satisfies the predictive-parity criterion forces false-positive and false-negative rates to differ, so the criteria conflict by construction.
Boundary
Scoring, screening and triage models whose errors fall unevenly on groups: credit, insurance, hiring, health.
Symptom or risk
Each criterion gives a different verdict, so the review turns into an argument over which number is the fairness metric, or the team optimises one and unknowingly worsens another.
Exceptions
Equal prevalence or a perfect classifier removes the conflict; a regulation may prescribe the criterion.
Trade-offs
One named criterion is easy to monitor but visibly leaves other gaps; record the choice and revisit it.
Pitfalls
Picking the criterion after seeing which one the model already passes.
Lessons and concepts
Claims and their sources
Chouldechova shows that when outcome prevalence differs across groups, the constraints of the predictive bias criterion force false negative and false positive rates to also differ.