LLMs, GenAI, RAG, agents, evaluation · insight-1501

Test system-versus-user conflicts directly instead of assuming the system prompt wins

Situation
A fare-quote assistant's system prompt forbids quoting refund amounts, while users and pasted emails tell it to ignore earlier instructions.
Action
Add a conflict category to the eval set: user turns and pasted documents that contradict a system rule, plus benign requests that only resemble a conflict. Report how often the system rule held per model version, and keep non-negotiable rules enforced in code as well.
Reason
Wallace et al. describe models giving developer instructions the same weight as untrusted text and propose training for priority, so priority is a learned, imperfect behaviour rather than a guarantee.
Boundary
Assistants whose policies, personas or data-handling rules live in the system role.

Symptom or risk

Higher-priority rules lose to lower-priority text in a share of conversations, and ordinary accuracy tests never exercise the conflict.

Single-turn tasks where no user-controlled text reaches the prompt.

Conflict cases are written by hand and need refreshing as attack phrasing changes.

Counting only refusals of obviously hostile prompts, which hides over-blocking of benign requests that resemble them.

Wallace et al. argue that prompt injections and jailbreaks succeed partly because models give system prompts the same priority as untrusted user and third-party text, and they train models to follow an instruction hierarchy (applied to GPT-3.5, with improved robustness including on unseen attack types).
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions