LLMs, GenAI, RAG, agents, evaluation · insight-0556
Compare two models on the same items with a paired analysis
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Model A and model B are run on the same 300 eval items, and their accuracies are compared as two independent proportions.
Action
Compute the per-item score difference, report its mean and standard error (or a paired bootstrap or McNemar test for binary outcomes), and inspect the items where the models disagree.
Reason
Item difficulty is shared across models, so paired differences remove that variance and give a tighter, valid comparison.
Boundary
Prompt A/B tests, model upgrades, fine-tune versus baseline decisions.
Symptom or risk
The unpaired test ignores that some items are hard for both models, so a real difference is missed or a noisy one is overstated.
Exceptions
If the two systems were run on different items, a paired analysis is impossible; rerun on a common set.
Trade-offs
Requires storing item-level results, which you need for error analysis anyway.
Pitfalls
Reporting only aggregate scores, which makes paired analysis and error review impossible later.
Lessons and concepts
Claims and their sources
Miller (an arXiv preprint, not peer reviewed) recommends analysing paired question-level differences when two models are evaluated on the same questions.