Introduced on day 45
RLHF (reinforcement learning from human feedback)
A post-training pipeline that fits a reward model to human pairwise preferences and then optimises the language model to maximise that reward while staying close to a reference model.
In detail
The usual stages are supervised fine-tuning, reward-model training and policy optimisation with a KL penalty. Failure can arise in the feedback (noisy or biased), the reward model (an exploitable proxy) or the optimisation; pushing a proxy reward too far lowers the true reward, and length and reduced diversity are known side effects.