LLMs, GenAI, RAG, agents, evaluation · insight-1633
Distil a small model on teacher outputs from your own traffic and judge it on the tail the teacher handles
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, ML engineer, Data scientist
- Industries
- aviation, retail
Summary: situation, action, reason, boundary
Situation
A large model gives good answers on a narrow, high-volume task and the team wants a small model trained on its outputs to cut cost.
Action
Sample prompts from real traffic stratified by intent and difficulty, generate teacher outputs, review a sample against ground truth, train the student, and evaluate it against both the teacher and human labels on a time-held-out tail slice. Keep an escalation path to the teacher.
Reason
Distillation transfers a large model's behaviour into a smaller one, as Hinton and colleagues describe for ensembles; the student can only reproduce what the teacher outputs, so teacher errors and gaps become training signal.
Boundary
Narrow tasks with abundant traffic and a checkable quality bar.
Symptom or risk
The student matches the teacher on common cases, fails on rare ones, and inherits the teacher's mistakes as if they were labels.
Exceptions
Open-ended tasks where no reference or check exists to compare the student with.
Trade-offs
Teacher generation costs money upfront, and the student must be retrained whenever the task or teacher changes.
Pitfalls
Evaluating the student only against the teacher's answers, which hides shared errors.
Lessons and concepts
Claims and their sources
Hinton et al. describe compressing the knowledge of a large model or ensemble into a single smaller model for deployment and report that distillation improved a heavily used commercial acoustic model.