LLMs, GenAI, RAG, agents, evaluation · insight-1511
Track refusals on a benign in-domain prompt set for every model or prompt change
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- aviation, cross-industry
Summary: situation, action, reason, boundary
Situation
A safety-tuned model update ships to an airline help assistant, and the tests cover only prompts that should be refused.
Action
Build a set of safe prompts that share vocabulary with unsafe ones, plus unsafe contrasts that should still be refused, and report the refusal rate on both sides per model and prompt version.
Reason
Rottger et al. built XSTest from 250 safe prompts in ten types and 200 unsafe contrasts to expose exaggerated safety, where models refuse clearly safe prompts that resemble unsafe ones.
Boundary
Any assistant behind a safety-tuned model or a moderation layer.
Symptom or risk
Legitimate requests such as how to kill a duplicate reservation are declined because the wording resembles unsafe ones, and the damage appears only in satisfaction scores.
Exceptions
Settings where over-refusal is explicitly preferred to risk; make that choice on purpose.
Trade-offs
Writing the prompts takes domain knowledge, and public sets do not cover your vocabulary.
Pitfalls
Driving the benign refusal rate down while no longer tracking the unsafe side.
Lessons and concepts
Claims and their sources
Rottger et al. present XSTest, 250 safe prompts across ten prompt types plus 200 unsafe contrast prompts, to identify exaggerated safety behaviours in which models refuse clearly safe prompts because of similar language to unsafe ones.