Introduced on day 45
DPO (direct preference optimization)
A preference-tuning method that trains the language model directly on chosen and rejected answers with a classification-style loss relative to a reference model, without a separate reward model or RL loop.
In detail
The loss equals log 2 when the policy matches the reference and falls as the policy raises the relative likelihood of the chosen answer. It is simpler to run than a reward-model plus PPO pipeline, but whether it matches tuned PPO is setup-dependent and contested, and it can be sensitive to shift between the preference data and the model's own outputs.