LLMs, GenAI, RAG, agents, evaluation · insight-0569

Measure output diversity after preference tuning when variety is part of the product

Situation
A preference-tuned model generates marketing copy for 300 routes, and the drafts start to sound alike.
Action
Add diversity measures to the eval (distinct n-grams, embedding spread across samples, duplicate rate) alongside quality, and compare the supervised and preference-tuned checkpoints before choosing one.
Reason
Research found RLHF improved out-of-distribution generalisation compared with supervised fine-tuning but substantially reduced output diversity.
Boundary
Creative generation, synthetic data, brainstorming, test-case generation.

Symptom or risk

Campaigns look repetitive, and synthetic data generated with the model covers fewer phrasings than expected.

Tasks that need one correct answer welcome reduced diversity.

Raising temperature restores some variety at a cost in quality and consistency.

Judging diversity from a handful of samples.

Kirk et al. found RLHF generalised better out of distribution than supervised fine-tuning but substantially reduced output diversity.
Understanding the Effects of RLHF on LLM Generalisation and Diversity