LLMs, GenAI, RAG, agents, evaluation · insight-0523
Balance and shuffle few-shot examples, then test several orders
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A few-shot prompt for classifying refund requests lists five approve examples followed by one reject example.
Action
Balance labels in the demonstrations, evaluate a few random orderings on a development split, report the spread, and keep an order that is robust rather than the luckiest one.
Reason
Studies found few-shot accuracy can range from near chance to strong depending on example choice and order, with biases toward frequent and recent labels.
Boundary
In-context classification and extraction with demonstrations.
Symptom or risk
Predictions lean toward the majority or most recent label, and accuracy swings when someone reorders the examples.
Exceptions
Zero-shot prompts and fine-tuned models are less exposed, though still worth a check.
Trade-offs
Testing orders multiplies evaluation runs.
Pitfalls
Selecting the best of many orders on the test set, which overfits the evaluation.
Lessons and concepts
Claims and their sources
Zhao et al. found few-shot accuracy varied from near chance to near state of the art with prompt format, chosen examples and their order, and named majority-label and recency biases.
Lu et al. found that the order of few-shot samples alone could separate near state-of-the-art from random-guess performance.