LLMs, GenAI, RAG, agents, evaluation · insight-1571
Record every tool request and response so a failed run can be replayed offline
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
An agent failed on a customer case last Tuesday, and the engineer cannot reproduce it because the live systems have changed.
Action
Log each tool call with its arguments and exact response, the model version and prompt version, then add a replay mode in which stubs return the recorded responses so the run can be re-executed deterministically against new code.
Reason
Live state drifts and model output varies, but recorded tool responses fix the environment so changes to prompts or logic can be compared on the same trajectory.
Boundary
Agents whose tools read changing data.
Symptom or risk
The failure cannot be reproduced or tested after a fix, so the same bug returns and fixes are guesses.
Exceptions
Records may contain personal data and need redaction and retention limits.
Trade-offs
Storage and privacy handling; the model's own output still varies between replays.
Pitfalls
Recording only the model's messages and not the tool results.
Lessons and concepts
Claims and their sources
LangGraph documentation lists time travel and fault tolerance among the uses of saved checkpoints, showing that replay from stored state is a supported pattern.