LLMs, GenAI, RAG, agents, evaluation · insight-0567
Track a gold metric while optimising against a reward model, and stop before it turns down
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
During RL fine-tuning, the reward-model score keeps rising run after run, and the team keeps training.
Action
Hold out a gold evaluation (human ratings or a stronger independent judge) and track it with the KL divergence from the starting model; keep the checkpoint where gold quality peaks, not where proxy reward peaks.
Reason
Research on reward-model overoptimisation found that optimising a proxy reward eventually lowers the gold reward, with the pattern depending on the method and on distance from the initial policy.
Boundary
RLHF, best-of-n sampling against a reward model, any optimisation against a learned metric.
Symptom or risk
Outputs exploit quirks of the reward model and get worse by human judgment, a Goodhart failure.
Exceptions
Objectives that are exactly verifiable, such as unit tests passing, are less exposed but can still be gamed.
Trade-offs
Gold evaluations are expensive, so they are run less often.
Pitfalls
Using the same reward model for training and for reporting results.
Lessons and concepts
Claims and their sources
Gao et al. found, in a setup where a fixed synthetic gold reward model stands in for human raters, that optimising against a proxy reward model eventually decreases the gold reward, with functional forms that depend on the optimisation method and the KL distance from the initial policy.