LLMs, GenAI, RAG, agents, evaluation · insight-0551
Judge a compressed model on your tasks, not on perplexity alone
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A self-hosted model is quantized or pruned to cut serving memory, and the team signs off because perplexity barely moved.
Action
Run the full task eval set, including knowledge-intensive and long-tail slices, on each compressed variant, and choose the most compressed variant that stays within the agreed quality tolerance.
Reason
Research found perplexity an inadequate measure of compressed models; on knowledge-intensive tasks pruning degraded performance even at modest sparsity, while quantization held up better but still needs task checks.
Boundary
Post-training quantization, pruning and distillation of open-weight models.
Symptom or risk
Knowledge-heavy answers, rare-entity facts or long-form reasoning quietly degrade, and the loss surfaces only in complaints.
Exceptions
Hosted APIs make this choice for you; compare their tiers on your eval set instead.
Trade-offs
Every variant costs an evaluation run; lower precision saves memory and often latency.
Pitfalls
Testing only the average score and missing a collapse on one slice.
Lessons and concepts
Claims and their sources
Jaiswal et al. found perplexity an inadequate measure for compressed LLMs; on knowledge-intensive tasks pruning degraded performance even at modest sparsity, while quantization held up better.