LLMs, GenAI, RAG, agents, evaluation · insight-0525

Evaluate prompts across several formatting variants before comparing models

Situation
Two candidate models are compared with one prompt template that was tuned on model A.
Action
Write a few meaning-preserving variants of the prompt format, run every model on all of them, and compare score ranges or each model's best variant rather than one template.
Reason
Research showed meaning-preserving formatting changes can move accuracy by large margins for some models, and recommended reporting a range across formats.
Boundary
Model selection, vendor comparisons, benchmark reports.

Symptom or risk

Model B loses because of incidental choices such as separators, casing or field labels, and the selection decision is wrong.

Once a model is chosen, freeze one format and version it.

More evaluation runs per comparison.

Tuning formats on the same test items used for the final comparison.

Sclar et al. found meaning-preserving formatting changes caused up to 76 accuracy points of difference for one model, and recommended reporting performance across a range of formats.
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting