LLMs, GenAI, RAG, agents, evaluation · insight-0638

Treat writes to long-term agent memory as untrusted input with provenance

Situation
An assistant saves "facts" or preferences from conversations, documents or web pages into persistent memory reused in later sessions.
Action
Store each memory with its source and timestamp, restrict what may be written (preferences, not instructions), let users view and delete entries, and never let memory content grant permissions.
Reason
Greshake and colleagues showed that instructions embedded in content an LLM application ingests can hijack its behaviour; persistent memory turns a one-off injection into a lasting one.
Boundary
Assistants with cross-session memory or shared team knowledge bases.

Symptom or risk

An instruction planted in a document gets stored as memory and shapes every future session, long after the source is gone.

Memory limited to structured user settings chosen in the interface.

Stricter memory rules make the assistant less adaptive.

Summarising untrusted pages into memory, which launders their origin.

Indirect prompt injection plants instructions in data that LLM-integrated applications retrieve or ingest, which can manipulate their behaviour.
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection