LLMs, GenAI, RAG, agents, evaluation · insight-0524
Use few-shot examples to show format and label space, not to teach new rules
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A team tries to teach the model a new internal overbooking-compensation rule by adding three worked examples to the prompt.
Action
State the rule explicitly in the instructions or supply it through retrieval, use demonstrations mainly to fix output format, label vocabulary and input style, and test on cases unlike the examples.
Reason
Research found that on classification and multiple-choice tasks, randomly replacing labels in demonstrations barely hurt performance for the models tested, suggesting demonstrations mainly convey label space, input distribution and format.
Boundary
In-context learning for classification and extraction.
Symptom or risk
Outputs copy the example format and labels but apply the rule inconsistently to new cases.
Exceptions
Some models may rely more on input-label mappings; confirm on your own eval set.
Trade-offs
Explicit rules make prompts longer.
Pitfalls
Evaluating only on cases that resemble the demonstrations.
Lessons and concepts
Claims and their sources
Min et al. found that randomly replacing demonstration labels barely hurt in-context learning on classification and multiple-choice tasks across 12 models, and that demonstrations mainly convey label space, input distribution and format.