LLMs, GenAI, RAG, agents, evaluation · insight-1504
Sweep the number of demonstrations against cost before settling on a shot count
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A team uses five demonstrations by habit, although the context window would hold hundreds of labelled fare-rule examples.
Action
Evaluate shot counts such as 5, 25, 100 and 400 on a development split with several orderings, plot accuracy and cost per request, and choose the knee; if many-shot wins, cache the static example block.
Reason
Agarwal et al. report significant gains moving from few-shot to many-shot on many tasks, while inference cost grows linearly with the number of shots, so the right count is an economic choice.
Boundary
Classification, extraction and translation-like tasks with abundant labelled data and a long-context model.
Symptom or risk
Accuracy is left on the table, or someone pastes hundreds of examples and pays for them on every request without measuring the gain.
Exceptions
High-volume tasks where fine-tuning is cheaper, and tasks that need fresh facts, which demonstrations do not supply.
Trade-offs
Long prompts raise latency and may themselves degrade with length.
Pitfalls
Choosing the shot count and the order on the test set.
Lessons and concepts
Claims and their sources
Agarwal et al. report significant performance gains from few-shot to many-shot in-context learning across generative and discriminative tasks, with inference cost growing linearly in the number of shots, and with frontier models benefiting to varying degrees.