LLMs, GenAI, RAG, agents, evaluation · insight-0534

Never draw few-shot demonstrations from the evaluation set

Situation
To improve a classification prompt quickly, an engineer copies the hardest misclassified eval cases into the prompt as examples.
Action
Keep a separate pool for demonstrations, check new demonstrations for overlap with eval items (exact and near-duplicate), and rerun evaluation on a held-out split after any demonstration change.
Reason
Copying test items into the prompt is test-set leakage in a new form; the score measures memorised cases rather than generalisation.
Boundary
Few-shot prompts, retrieved-example prompts, prompt-optimisation tools.

Symptom or risk

The eval score jumps because those items are now answered from the prompt, and the reported gain does not appear in production.

Using eval failures to understand error types is fine; write new, similar examples rather than copying.

Maintaining two pools takes some discipline and storage.

Automatic prompt optimisers that select demonstrations using the same set they report on.