LLMs, GenAI, RAG, agents, evaluation · insight-0530
Prune long system prompts by ablating rules against the eval set
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI
- Industries
- aviation, telecom
Summary: situation, action, reason, boundary
Situation
A customer-service system prompt has grown to dozens of rules added after individual complaints, some of them contradicting each other.
Action
Remove or merge rules one group at a time and rerun the eval set; keep rules whose removal hurts a measured case, document why each remaining rule exists, and link it to a test.
Reason
Each rule should earn its place with evidence; untested rules accumulate cost, conflicts and maintenance risk without a measured benefit.
Boundary
Mature prompts edited by several people over months.
Symptom or risk
Nobody knows which rules still matter, new rules interact unpredictably, and every edit risks a regression.
Exceptions
Rules required by policy or law stay even if no eval case triggers them; add a case that does.
Trade-offs
Ablation needs repeated evaluation runs.
Pitfalls
Removing a rule because the eval set has no case for it, rather than adding the missing case.