LLMs, GenAI, RAG, agents, evaluation · insight-0522
Write success criteria and test cases before the first prompt revision
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Applied AI, Data scientist, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A team has rewritten a booking-change prompt eleven times, each time reacting to whichever example last went wrong.
Action
Define measurable success criteria, collect roughly 50-100 representative and edge cases from real traffic with expected outputs, score the current prompt as the baseline, and accept a change only if it improves the suite without new regressions.
Reason
Vendor prompt-engineering guidance places success criteria and empirical tests before prompt iteration, because without them improvement cannot be measured.
Boundary
Every prompt that will run in production.
Symptom or risk
Each fix breaks earlier cases nobody rechecks, and nobody can say whether version 11 beats version 1.
Exceptions
Throwaway exploration can skip this; anything shipped cannot.
Trade-offs
A day of case collection before visible progress.
Pitfalls
Writing cases only from the happy path or from the prompt author's imagination.
Lessons and concepts
Claims and their sources
Anthropic's prompt-engineering overview advises defining success criteria and ways to test against them empirically before iterating on prompts.