LLMs, GenAI, RAG, agents, evaluation · insight-0663
Version prompt, retrieval settings and model together as one release
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- ML engineer, Applied AI, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Prompts live in code, retrieval parameters in a config service and the model name in an environment variable, each changed independently.
Action
Bundle prompt templates, model id, decoding settings, retrieval and chunking parameters, and index version into one versioned configuration; stamp its id on every trace and evaluate it as a unit.
Reason
LLM behaviour depends on the whole configuration, not on any single file. A single release id makes evaluation results, incidents and rollbacks refer to the same thing.
Boundary
Any team shipping LLM features more than once a month.
Symptom or risk
Quality shifts after a deploy and nobody can reconstruct which combination served the bad answers, so rollback is guesswork.
Exceptions
Early prototypes with one developer, though even then a commit hash per run helps.
Trade-offs
Small prompt edits now go through a release process; keep it lightweight.
Pitfalls
Versioning the prompt but rebuilding the index in place, so the same release id points at different documents.