LLMs, GenAI, RAG, agents, evaluation · insight-1603
Restrict tool access for the rest of a session once untrusted text has entered the context
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
One long agent session reads external content such as email, web pages or tickets and also holds state-changing tools.
Action
Track provenance on the context. After any untrusted source is loaded, switch the session to a stricter policy: read-only tools, or write tools only with human approval and with arguments not derived from the untrusted text. Reset the policy only by starting a new session.
Reason
Beurer-Kellner and colleagues describe the principle that once an agent has ingested untrusted input it must be constrained so that this input cannot trigger consequential actions. A dynamic policy enforces that outside the model.
Boundary
Agents that mix reading untrusted content with actions in one session.
Symptom or risk
A static allow-list lets the agent call the send or refund tool after it read a hostile page, because the call itself looks permitted.
Exceptions
Sessions whose sources are all internal and reviewed.
Trade-offs
Read-then-act tasks need an approval step or must be split into two sessions.
Pitfalls
Marking only the first message as untrusted, and forgetting that a summary of untrusted text carries the same taint.
Lessons and concepts
Claims and their sources
The design-patterns paper states that once an agent has ingested untrusted input it must be constrained so that the input cannot trigger consequential actions.