LLMs, GenAI, RAG, agents, evaluation · insight-1503
Pick few-shot examples by similarity to each request and keep the fixed set as the baseline
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A classifier for passenger messages uses the same six hand-picked examples for every request, and accuracy is uneven across message types.
Action
Embed a labelled pool, retrieve the nearest examples for each message, and compare with the fixed set on a held-out split sliced by message type; log which examples were retrieved so failures can be reproduced.
Reason
Liu et al. found that retrieving semantically similar examples beat random selection across several benchmarks with GPT-3, and that encoders fine-tuned on task data retrieved even more helpful examples.
Boundary
Varied input spaces with a labelled pool of a few hundred cases or more.
Symptom or risk
Rare message types get no relevant demonstration, and extra examples added to cover them dilute the prompt without helping.
Exceptions
Narrow tasks where a fixed set already covers the inputs, since selection then only adds variance.
Trade-offs
An extra retrieval step, an index to maintain, and outputs that now depend on the pool contents.
Pitfalls
Drawing the pool from the evaluation set, which turns the measured gain into leakage.
Example
Hypothetical: a message about a missing wheelchair is shown three earlier wheelchair cases as demonstrations instead of generic baggage ones.
Lessons and concepts
Claims and their sources
Liu et al. report that selecting in-context examples by semantic similarity to the test sample consistently outperformed random selection on several NLU and generation benchmarks with GPT-3, with larger gains when the sentence encoder was fine-tuned on task-related data.