LLMs, GenAI, RAG, agents, evaluation · insight-0623
Run RAG ablations one component at a time with per-stage metrics
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A new RAG version changes the chunker, the embedding model and the prompt at once, and end-to-end accuracy improves.
Action
Change one component per run, hold the golden set and other settings fixed, and record recall@k, reranked MRR, faithfulness and answer correctness for each run in an experiment log.
Reason
Per-stage metrics show where each change acts: a chunking change should move retrieval metrics first, a prompt change should move only answer metrics.
Boundary
Day 51 optimisation work and any release that touches retrieval.
Symptom or risk
You cannot tell which change helped, and a harmful change hides behind a helpful one until the next release.
Exceptions
Components that interact strongly, such as chunk size and k, may need a small grid rather than one-at-a-time runs.
Trade-offs
Sequential ablations take longer; batch offline evaluation runs to keep cost down.
Pitfalls
Re-tuning on the golden set after every ablation until it stops being a fair test; keep a held-out slice.