Engineering, production, governance, cost, communication · insight-0276
Instrument a model service with the golden signals plus model-specific signals
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model endpoint has a health check that returns OK, and users report slow and odd recommendations.
Action
Emit latency percentiles, traffic, error rate and saturation for the service, plus prediction distribution, fallback rate, feature null rate and model version per request; trace requests across feature lookups and the model call.
Reason
Latency, traffic, errors and saturation are the core user-facing signals for any service; model signals add the ML-specific failure modes.
Boundary
Online inference services, LLM endpoints, feature servers.
Symptom or risk
Liveness checks do not reveal latency spikes, saturation, error bursts or a sudden shift in score distribution.
Exceptions
Batch jobs need run duration, row counts and completion time instead of request latency.
Trade-offs
High-cardinality labels (per customer) make metrics expensive; use traces or logs for those.
Pitfalls
Averaging latency instead of tracking tail percentiles.
Lessons and concepts
Claims and their sources
The four golden signals of monitoring are latency, traffic, errors and saturation.