LLMs, GenAI, RAG, agents, evaluation · insight-0571

Re-measure core capabilities after alignment tuning to catch an alignment tax

Situation
A model tuned for helpfulness and tone on support conversations is promoted after winning preference comparisons.
Action
Run the capability eval (task metrics and general canary sets) before and after every alignment step, set a maximum acceptable regression per capability, and consider mixing original-task data into tuning when regressions appear.
Reason
The InstructGPT work observed performance regressions on some public NLP datasets after RLHF (an "alignment tax") and reduced them, without fully removing them, by mixing pretraining updates into PPO.
Boundary
Any preference or instruction tuning of a model used for several tasks.

Symptom or risk

Extraction accuracy, arithmetic or domain Q&A quietly regress, because the preference eval did not cover them.

Single-purpose models only need their own task eval.

Mixing data adds tuning complexity and may slow alignment gains.

Evaluating aligned models only with preference win rates.

Ouyang et al. report performance regressions on some public NLP datasets after RLHF (an "alignment tax") and reduce them, though not completely, by mixing pretraining gradients into PPO (PPO-ptx).
Training language models to follow instructions with human feedback