LLMs, GenAI, RAG, agents, evaluation · insight-0539
Name the missing capability before choosing prompting, retrieval or fine-tuning
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, Data scientist, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A stakeholder asks for the model to be fine-tuned on company data because answers about baggage policy are wrong.
Action
Inspect failures and classify each as missing knowledge (retrieval), unclear instructions (prompting) or inconsistent behaviour, format or tone (fine-tuning), then address the dominant class first and re-measure.
Reason
Vendor guidance frames retrieval as supplying missing or proprietary context and fine-tuning as shaping consistent behaviour, with prompting and an eval set as the starting point.
Boundary
Every adaptation request for an LLM feature.
Symptom or risk
Fine-tuning is chosen for a knowledge problem, so answers stay outdated and the team has a model to retrain whenever policy changes.
Exceptions
Combined problems need combined solutions; the methods are additive.
Trade-offs
Failure classification takes reviewer time before any build starts.
Pitfalls
Classifying failures from a few anecdotes rather than a labelled sample.
Lessons and concepts
Claims and their sources
OpenAI's accuracy-optimisation guide frames retrieval as context optimisation for missing knowledge and fine-tuning as optimising consistent behaviour, recommends starting with prompt engineering and an evaluation set, and calls the techniques additive.