LLMs, GenAI, RAG, agents, evaluation · insight-0529
Repair failed outputs with the validator's error message and a retry cap
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A structured extraction fails business-rule validation on a few percent of documents, and an engineer adds code that silently coerces bad values into defaults.
Action
On validation failure, re-prompt once or twice with the original input, the failed output and the specific validator error; if it still fails, mark the record as failed and send it to review; log attempt counts.
Reason
A concrete error message is external feedback the model can act on, while coercion destroys the signal that something went wrong.
Boundary
Extraction and transformation pipelines with automatic validators.
Symptom or risk
Coerced defaults look like real data, hide the failure rate and corrupt downstream analytics.
Exceptions
When failures cluster on one input type, fix the prompt or schema instead of relying on retries.
Trade-offs
Each retry adds latency and cost; caps keep them bounded.
Pitfalls
Retrying without telling the model what was wrong, which often reproduces the same output.