LLMs, GenAI, RAG, agents, evaluation · insight-1597

Blind and shuffle annotation tasks, and seed known-answer items to catch drift

Situation
Reviewers compare outputs from two systems, with system names visible and a fixed order, over several days of labelling.
Action
Hide system identity, randomise the order and position of outputs, and seed 5 to 10 percent of the queue with items of known verdict; review each annotator's accuracy on seeded items by day and re-calibrate or exclude batches that fall.
Reason
Blinding removes the source cue and seeded items give a running measure of annotator reliability that does not depend on peers.
Boundary
Human preference or rubric evaluations spanning many sessions or annotators.

Symptom or risk

Brand expectations, order effects and fatigue leak into labels, and nobody can tell when a reviewer's standard drifted.

Seeds are unnecessary for a single short session by one trained expert.

Seeding consumes labelling budget; the seeds must not be recognisable.

Reusing the same seeds so annotators learn them.

Chatbot Arena presents two anonymous model answers in randomised battles and reveals model identities only after the vote.
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference