LLMs, GenAI, RAG, agents, evaluation · insight-1704
Pin and publish the eval harness version, because the same model scores differently across implementations
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A team compares its own benchmark number for an open model with a published figure, or moves an eval from one framework to another.
Action
Record the harness name, commit, prompt template, shot count and answer-extraction rule next to every score. Before comparing with a published number, reproduce it with the same implementation, and rank candidates only within one harness version.
Reason
A 2023 Hugging Face analysis ran one benchmark through three codebases that differ in prompt wording and in how an answer is scored, and got different scores and orderings for the same models.
Boundary
Public-benchmark reproduction, vendor comparisons and migrations between evaluation frameworks.
Symptom or risk
A gap of many points is blamed on the model or the checkpoint when prompt wording and answer scoring differ, and rankings can change with the harness.
Exceptions
A private eval run only by your own code, which is its own fixed definition.
Trade-offs
Reproduction runs cost compute, and a pinned harness lags upstream fixes.
Pitfalls
Comparing a number from a blog post with a number from your harness, or mixing harness versions in one table.
Example
From the post (MMLU, HELM / Harness / Original): llama-65b 0.637 / 0.488 / 0.636, a 14.9-point gap; falcon-40b scores 0.527 against llama-30b's 0.457 under the harness but 0.571 against 0.583 under HELM, so their order flips.
Lessons and concepts
Claims and their sources
The Hugging Face post (June 2023) reports MMLU scores for eight models under three implementations, for example llama-65b at 0.637, 0.488 and 0.636, and attributes the differences to prompt wording and to how the answer is compared (letter probabilities, next-token match or full-answer likelihood); it says the harness was later updated to match the original.