Engineering, production, governance, cost, communication · insight-1719
Keep unbounded identifiers out of metric labels and log them as events instead
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A team adds customer id or request id as a label on a model-monitoring metric to see one user's score trend.
Action
Label metrics only with bounded dimensions (model version, route, segment bucket) and count the distinct values per label in review. Send per-entity detail to logs or a trace store and query it there.
Reason
Prometheus documents that each unique combination of label values is a new time series and cautions against high-cardinality labels such as user ids; aggregates need only a few series.
Boundary
Time-series metric stores with per-label series, such as Prometheus-style systems and many hosted equivalents.
Symptom or risk
Every new value creates a new time series, so storage, query time and cost grow with traffic, and the backend slows during the incident the dashboard was meant for.
Exceptions
A small, fixed set of entities, such as 30 named customers on a contract, may be labelled.
Trade-offs
Per-entity questions need a second tool, and sampled logs may miss rare entities.
Pitfalls
Hashing ids into too many buckets; adding a label to a metric that already carries several.
Example
Hypothetical arithmetic: 5 model versions x 12 routes x 20,000 daily active customers is 1,200,000 series for one metric, against 60 without the customer label.
Lessons and concepts
Claims and their sources
The Prometheus naming guide states that every unique combination of label key-value pairs represents a new time series and cautions against labels holding high-cardinality values such as user ids or email addresses.