LLMs, GenAI, RAG, agents, evaluation · insight-1620
Measure guardrail false positives on benign traffic, because rare attacks make most blocks wrong
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, Applied AI, ML engineer
- Industries
- telecom, cross-industry
Summary: situation, action, reason, boundary
Situation
A classifier blocks prompts it believes are attacks or policy violations, and its quality is reported as recall on a balanced test set.
Action
Evaluate on a sample of real benign traffic plus benign look-alikes (safe prompts that resemble unsafe ones). Report false positives per 1,000 requests and precision at the expected base rate, choose the threshold from the cost of a wrongly blocked request against a missed attack, and slice by language and feature.
Reason
Axelsson showed that at low base rates the false-alarm rate limits a detector's usefulness. XSTest was built because models also refuse safe prompts that merely look unsafe.
Boundary
Input and output classifiers, moderation endpoints and injection detectors.
Symptom or risk
In production attacks are rare, so most blocks hit legitimate users, support tickets rise and the team switches the guardrail off.
Exceptions
Hard-block categories where a miss is catastrophic and a human reviews every block.
Trade-offs
A lower false-positive rate usually costs recall; make the exchange rate explicit.
Pitfalls
Reporting accuracy, and tuning on a test set built only from attack strings.
Example
Hypothetical: 1,000,000 requests a day, 0.1% attacks, recall 90%, false-positive rate 1%. That is 900 attacks caught and 9,990 benign blocks, so only 8.3% of blocks are attacks.
Lessons and concepts
Claims and their sources
Axelsson argues that because of the base-rate fallacy the false alarm rate limits the usefulness of an intrusion detector.
XSTest has 250 safe prompts in ten types and 200 unsafe contrast prompts to measure exaggerated safety behaviour, meaning refusals of safe requests.