LLMs, GenAI, RAG, agents, evaluation · insight-1599
Report attack success rate together with the number of attempts the attacker was allowed
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A red-team summary says "the model resisted 98% of attacks" after one attempt per prompt.
Action
Report attack success as a function of attempt budget (1, 10, 100 samples) with the perturbation types used, and compare defences at equal budgets.
Reason
Success under repeated random augmentation was reported to rise as a power-law-like function of the number of samples, so a one-shot rate is a lower bound.
Boundary
Safety and prompt-injection evaluations of black-box assistants.
Symptom or risk
A single try understates risk, because an attacker can resample and perturb a prompt many times and needs only one success.
Exceptions
Rate-limited or authenticated deployments may bound the realistic budget, which is itself a stated assumption.
Trade-offs
Many-sample evaluation is costly; sample budgets geometrically.
Pitfalls
Extrapolating the published numbers to your model; they are specific to models tested in late 2024.
Lessons and concepts
Claims and their sources
Hughes et al. report that attack success rate under Best-of-N sampling of augmented prompts follows power-law-like behaviour as a function of the number of samples, reaching 89% on GPT-4o and 78% on Claude 3.5 Sonnet at 10,000 samples.