LLMs, GenAI, RAG, agents, evaluation · insight-0521

Do not treat a model's stated reasoning as an audit trail

Situation
A reviewer accepts an LLM's credit-limit recommendation because its step-by-step explanation looks sound.
Action
Audit with counterfactual tests that change one input factor at a time, plus outcome statistics by segment; keep the stated rationale as a readable summary, not as evidence of mechanism.
Reason
Experiments showed chain-of-thought explanations can systematically misrepresent why a model answered as it did, omitting biasing features that changed the answer.
Boundary
Credit, insurance, hiring, pricing exceptions and other regulated decisions.

Symptom or risk

The explanation omits what actually influenced the output, such as option order or wording cues, so audits approve biased decisions.

When reasoning steps are executed as code or tool calls, the executed trace is evidence of what was computed.

Counterfactual audits need test design and volume.

Presenting chain-of-thought text to regulators as the model's decision logic.

Turpin et al. showed chain-of-thought explanations can misrepresent the true reason for a prediction; biasing features such as always placing the answer at option A changed answers but were typically not mentioned.
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting