Engineering, production, governance, cost, communication · insight-1287
Design a contest path into decisions that may fall under solely automated processing
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Data scientist, Applied AI
- Industries
- banking, insurance, public-sector
Summary: situation, action, reason, boundary
Situation
A team plans to reject consumer loan applications straight from a score with no reviewer step (hypothetical, EU applicants).
Action
At design time classify each automated decision by its effect on the person. For significant ones, build the review queue, the information shown to the reviewer and the contest route into the workflow, and give reviewers the authority and data to change the outcome.
Reason
GDPR Article 22 gives people the right not to be subject to solely automated decisions with legal or similarly significant effects, and for the contract-necessity and explicit-consent exceptions the safeguards include at least human intervention, expressing a view and contesting the decision.
Boundary
Credit, insurance, hiring, benefits and account-closure decisions affecting people.
Symptom or risk
Decisions with legal or similarly significant effect need safeguards such as human involvement and a way to contest, and bolting them on after launch costs a redesign.
Exceptions
Whether a process is solely automated or significant is a legal judgement; take it to the privacy owner, and other jurisdictions differ.
Trade-offs
Review capacity limits throughput; send adverse and uncertain cases to people.
Pitfalls
A reviewer who sees only the score and cannot change the outcome.
Lessons and concepts
Claims and their sources
GDPR Article 22 grants the right not to be subject to solely automated decisions with legal or similarly significant effects; for the contract-necessity and explicit-consent exceptions the safeguards include at least human intervention, expressing a point of view and contesting the decision (Article 22(3)); the law-authorised exception needs suitable measures.