LLMs, GenAI, RAG, agents, evaluation · insight-1512
Diff output shape between model versions on a fixed prompt suite before switching traffic
- Evidence type
- empirical finding
- Difficulty
- easy
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A provider releases a successor model, and a JSON-extraction prompt for disruption notices is moved to it after a check on accuracy alone.
Action
Run both versions on the same suite and record parse success, length distribution, presence of fences or prose, refusal rate and field-level agreement; fail the migration on shape regressions as well as on accuracy.
Reason
Chen, Zaharia and Zou compared March and June 2023 snapshots of GPT-3.5 and GPT-4 and reported substantial behaviour changes, including more formatting mistakes in code generation and weaker instruction following for GPT-4.
Boundary
Any pipeline that parses model text.
Symptom or risk
Accuracy holds but the wrapper changes: code fences, a preamble sentence, longer answers or different refusals, and the downstream parser breaks or truncates.
Exceptions
Outputs read only by humans, where shape changes cost little.
Trade-offs
One suite run per model release, cheap next to an incident.
Pitfalls
Comparing only aggregate accuracy, which can stay flat while output shapes change.
Lessons and concepts
Claims and their sources
Chen, Zaharia and Zou report that GPT-3.5 and GPT-4 behaviour changed substantially between March and June 2023 snapshots on several tasks, that both made more formatting mistakes in code generation in June, and that GPT-4's instruction following declined.