LLMs, GenAI, RAG, agents, evaluation · insight-0537
Require a fine-tune to beat the best prompt on a frozen eval set before you own it
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, ML engineer, Data scientist
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A team proposes fine-tuning a model for disruption-notice drafting after two weeks of prompt work.
Action
Freeze the eval set, record the strongest prompt plus few-shot baseline, estimate the full ownership cost of the fine-tune, and adopt it only if it beats the baseline by a margin larger than eval noise on the metrics that matter.
Reason
A fine-tuned model is a maintained asset; the decision should rest on measured improvement against the cheapest adequate alternative.
Boundary
Any proposal to fine-tune for quality, cost or latency.
Symptom or risk
The fine-tune ships with a small or unmeasured gain, and the team now owns training data, retraining, a model version and a new failure surface.
Exceptions
When the goal is a smaller, cheaper model at equal quality, compare on cost and latency too, not quality alone.
Trade-offs
A strong baseline takes effort that may make the fine-tune unnecessary, which is a good outcome.
Pitfalls
Comparing the fine-tune against an early, weak prompt.