LLMs, GenAI, RAG, agents, evaluation · insight-1540
Measure index freshness lag from source change to searchable, and show the as-of date in answers
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- ML engineer, Applied AI
- Industries
- aviation, banking
Summary: situation, action, reason, boundary
Situation
A fare-conditions PDF is replaced at 10:00, and the support assistant quotes the old rule that afternoon.
Action
Stamp each document version with its source-modified time, record when its chunks became searchable, alert on the lag percentile, and include the document's effective or as-of date next to cited claims.
Reason
Freshness is a pipeline property that can be measured; without a number it is discovered through customer complaints.
Boundary
Any RAG system over documents that change: policies, price lists, runbooks, regulations.
Symptom or risk
Ingestion queues, parser retries and embedding backlogs create an unmeasured gap, so the assistant is confidently stale and nobody can say for how long.
Exceptions
Static archives that never change.
Trade-offs
Tight lag targets increase ingestion frequency, load and cost.
Pitfalls
Measuring lag from ingestion start instead of the source's modified time, hiding the time spent waiting in queues.
Example
Synthetic: a document changes at 10:00 and becomes searchable at 14:30, a lag of 4.5 hours; a 1-hour p95 target would have paged someone.