LLMs, GenAI, RAG, agents, evaluation · insight-1587
Cluster the standard errors when eval questions share a document, ticket or customer
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
An eval has 300 questions written from 30 documents, ten per document, and you compute a score interval as if the 300 were independent.
Action
Treat the document as the sampling unit: bootstrap over documents or use clustered standard errors, and report the cluster count beside the question count.
Reason
The effective sample size is closer to the number of clusters than to the number of questions when items within a cluster tend to be right or wrong together.
Boundary
Reading-comprehension-style evals, multi-question tickets, multilingual copies of one item.
Symptom or risk
Questions about the same document are correlated, so the naive interval is too narrow and small differences look significant.
Exceptions
Independently sampled single questions need no clustering.
Trade-offs
Wider intervals, and fewer clusters than hoped.
Pitfalls
Paired model comparisons are also clustered; cluster the paired differences too.
Lessons and concepts
Claims and their sources
Miller reports that for Anthropic models clustered standard errors can be over three times larger than naive ones (about 3.0x on DROP, 1.1x on RACE-H, 1.9x on MGSM), and recommends clustering when questions are drawn in groups.