LLMs, GenAI, RAG, agents, evaluation · insight-0627
Monitor the ingestion pipeline, because parse failures drop evidence silently
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI, Data analyst
- Industries
- insurance, energy
Summary: situation, action, reason, boundary
Situation
A RAG index is rebuilt nightly from PDFs, HTML and spreadsheets owned by other teams.
Action
Track per-run counts of documents, chunks per document, empty or very short chunks, and parse errors by source; alert on sharp changes and keep a few golden documents whose extracted text is diffed every run.
Reason
Retrieval can only find what ingestion produced. Treating ingestion as a data pipeline with contracts catches failures that answer-quality metrics show late and vaguely.
Boundary
Any corpus refreshed automatically from sources you do not control.
Symptom or risk
A layout change makes the parser emit empty text or scrambled tables for a document family, and answers degrade weeks before anyone links it to ingestion.
Exceptions
Small static corpora that are ingested once and reviewed by hand.
Trade-offs
Alert thresholds need tuning to avoid noise from legitimate document growth.
Pitfalls
Monitoring only query latency and errors, which stay green while content disappears.