LLMs, GenAI, RAG, agents, evaluation · insight-1647

Alert on shifts in output-side signals, because a degrading assistant raises no exceptions

Situation
Dashboards show request rate, error rate and latency, all green, while answers have become shorter, more hedged or ungrounded.
Action
Track per route: finish-reason mix, output-length percentiles, refusal and abstain rate, empty-retrieval share, schema-failure rate and tool calls per task. Baseline them, and page only on sharp, user-affecting shifts; route slow drift to a review dashboard.
Reason
Latency and error signals measure the plumbing; the SRE monitoring guidance asks that pages be urgent and actionable, which is why output-quality proxies need baselines and thresholds rather than raw noise.
Boundary
Production assistants and extraction pipelines with stable traffic mix.

Symptom or risk

Quality degradation from a prompt, retrieval or model change is found by users or by a quarterly review.

Very low traffic, where counts are too small and sampled human review is a better signal.

Proxy signals can move for benign reasons such as a new customer cohort; confirm with sampled review.

Alerting on every metric, which trains the team to ignore pages.

The SRE book says pages should be urgent and actionable and that noisy alerts get ignored, with non-urgent issues better placed on dashboards; it names latency, traffic, errors and saturation as four golden signals.
Monitoring Distributed Systems (Site Reliability Engineering, chapter 6)