LLMs, GenAI, RAG, agents, evaluation · insight-0655
Keep a prompt-injection regression suite and report attack success next to task utility
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Security testing of an LLM feature was a one-off manual red-team session before launch.
Action
Turn red-team findings into automated cases (direct and indirect, through documents, emails and tool results), run them in CI, and track both attack success rate and normal task success for each defence.
Reason
AgentDojo scores utility alongside attack success because models already fail many tasks without any attack, and attacks break some security properties but not others. OWASP recommends regular adversarial testing.
Boundary
Any assistant with tools, retrieval over external content or sensitive data.
Symptom or risk
A later prompt, model or tool change reopens an injection path, and nobody notices because the suite that found it was never rerun.
Exceptions
Closed internal tools over trusted text, which still need a smaller suite.
Trade-offs
A fixed suite goes stale against adaptive attackers; refresh it with periodic manual red teaming.
Pitfalls
Declaring a defence effective because the attack rate fell, without checking that utility did not collapse too.
Lessons and concepts
Claims and their sources
AgentDojo evaluates prompt injection attacks and defences for tool-using agents on 97 tasks and 629 security test cases, measuring utility as well as attack success.
OWASP's Prompt Injection entry recommends regular adversarial testing and attack simulations.