LLMs, GenAI, RAG, agents, evaluation · insight-1594
Read thumbs-up and thumbs-down rates as a selection-biased signal and sample the silent majority
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- saas, e-commerce
Summary: situation, action, reason, boundary
Situation
A dashboard tracks the thumbs-down rate of an assistant and the team treats it as the quality metric.
Action
Use explicit feedback to find candidates for review, but estimate quality from a random sample of all sessions that a labeller or validated judge grades, and watch the response rate as its own metric.
Reason
The users who rate are not a random sample of the users served, and the rating mechanism itself changes over time.
Boundary
Chat assistants and search products with optional feedback buttons.
Symptom or risk
Only a small, self-selected fraction of sessions leave a rating, so the rate moves with UI changes, user mix and mood rather than with quality.
Exceptions
Forced-choice feedback at a mandatory step has a response rate near 100% and is much less selected.
Trade-offs
Random sampling costs labelling effort.
Pitfalls
Optimising the thumbs rate by hiding the button or asking at easy moments.
Example
Hypothetical: 2% of sessions are rated; a redesign moves the button and the response rate to 4% while thumbs-down share falls, with no change in graded quality.