LLMs, GenAI, RAG, agents, evaluation · insight-0566
Locate RLHF failures in feedback, reward model or policy before changing the recipe
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
After preference tuning, an internal assistant becomes evasive on some topics and overconfident on others, and the team considers switching alignment methods.
Action
Audit each stage separately, annotator agreement and guideline clarity, reward-model accuracy on held-out preference pairs, and policy behaviour on the eval set, and fix the weakest stage first.
Reason
A survey of RLHF groups its open problems and fundamental limitations by human feedback, reward model and policy, which gives a diagnostic map.
Boundary
Teams running preference tuning (RLHF or DPO) on their own data.
Symptom or risk
The method is swapped while the real cause, such as inconsistent annotator guidelines or a reward model that misreads the task, stays in place.
Exceptions
Teams using hosted models cannot inspect these stages; they test behaviour and choose providers instead.
Trade-offs
Stage-level audits need labelled preference data and time.
Pitfalls
Treating alignment as a single switch that is on or off.
Lessons and concepts
Claims and their sources
Casper et al. survey open problems and fundamental limitations of RLHF across human feedback, the reward model and the policy.
Ouyang et al. describe the RLHF pipeline as supervised fine-tuning, reward-model training on human comparisons, and policy optimisation with PPO.