LLMs, GenAI, RAG, agents, evaluation · insight-1583
Treat the judge model and prompt as part of the metric, and re-baseline when either changes
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
The provider updates the judge model, or someone edits the judge prompt, and eval scores shift without any change to your system.
Action
Pin a dated judge version, version the judge prompt, and keep an anchor set of frozen outputs with human labels; rerun the anchor set on every judge change and re-score history if the verdicts move.
Reason
The measuring instrument changed. Hosted models are known to behave differently across versions, so a fixed model name is not a fixed instrument.
Boundary
Any longitudinal eval that uses a hosted LLM as a judge.
Symptom or risk
A score drop or gain is attributed to your release when it comes from the grader, so trends across weeks are not comparable.
Exceptions
One-off comparisons scored in a single batch share one judge by construction.
Trade-offs
Pinned versions are retired on a schedule, so migration needs planning, plus the cost of re-scoring.
Pitfalls
Comparing a new run to scores stored from the old judge without re-scoring the old outputs.
Lessons and concepts
Claims and their sources
Chen, Zaharia and Zou found substantial behaviour changes between March and June 2023 versions of GPT-3.5 and GPT-4 on several tasks and recommended continuous monitoring.