LLMs, GenAI, RAG, agents, evaluation · insight-0508
Match checkpoint type and chat template to the job before judging an open model
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
An engineer downloads open model weights, sends plain instructions and gets rambling, off-task text.
Action
Confirm whether the checkpoint is base or instruction-tuned, format conversations with the model's own chat template (for example apply_chat_template in Transformers), version that template with the prompt, and only then run the eval set.
Reason
Base models are trained to continue text; instruction-tuned models were further trained to follow instructions in a specific format with control tokens, and wrong control tokens degrade output.
Boundary
Self-hosted open-weight models, fine-tuning pipelines, offline benchmarking.
Symptom or risk
The model is rejected as weak when the real problem is a base checkpoint or a prompt that ignores the model's conversation format.
Exceptions
Continued pretraining and raw text completion deliberately use the base checkpoint without a template.
Trade-offs
Little cost; this is setup hygiene that belongs in the prompt version.
Pitfalls
Fine-tuning with one template and serving with another.
Lessons and concepts
Claims and their sources
Ouyang et al. showed that fine-tuning with human feedback made models follow instructions better than the base GPT-3 model, with outputs of a much smaller tuned model preferred by labellers.
The Transformers documentation states that models expect different chat formats and control tokens and that using the wrong ones can make performance drastically worse.