LLMs, GenAI, RAG, agents, evaluation · insight-1521
Compress prompts only after checking that identifiers, numbers and negations survive
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- insurance, cross-industry
Summary: situation, action, reason, boundary
Situation
A cost review proposes a prompt-compression library to shrink long retrieved policy text by a factor of five.
Action
Compress a sample, verify by script that protected tokens (codes, numbers, dates, negations) survive or are exempted, compare end-task accuracy at several ratios, and choose the largest ratio inside your tolerance.
Reason
LLMLingua reports up to 20x compression with little performance loss on the authors' four datasets; token-level pruning is lossy by design, and exact values are where a loss matters most.
Boundary
Long narrative context in cost-sensitive pipelines.
Symptom or risk
Narrative context compresses well while fare codes, amounts, dates and the word not are dropped or altered, and answers turn wrong with no format signal.
Exceptions
Context in which every token is exact, such as tables, codes and contract clauses.
Trade-offs
A compression model call and added latency, partly offset by shorter prompts.
Pitfalls
Compressing a different way on each request, which changes the prefix and defeats prompt caching.
Lessons and concepts
Claims and their sources
Jiang et al. report that LLMLingua allows up to 20x prompt compression with little performance loss on GSM8K, BBH, ShareGPT and Arxiv-March23, using a budget controller and token-level iterative compression.