LLMs, GenAI, RAG, agents, evaluation · insight-0673

Pick model configurations from the quality-cost frontier, not by model size

Situation
Model selection asks "which is the best model?" and defaults to the largest one available.
Action
Evaluate candidate configurations (model, context size, retrieval k, prompting) on the same golden set, plot task quality against cost per request and p95 latency, and choose among points that no other point beats on all axes.
Reason
Model prices varied widely, by up to two orders of magnitude in the 2023 FrugalGPT survey, while quality differences on a narrow task can be small; only a measured frontier shows where extra spend buys quality.
Boundary
Initial model choice, renewals, and every provider price change.

Symptom or risk

The team pays for quality the task does not need, or misses a cheaper configuration that matches quality once retrieval or prompting is fixed.

Tasks with a hard quality floor (regulated advice) where only one configuration qualifies.

Frontier points shift with traffic mix and prices; refresh quarterly.

Comparing configurations evaluated on different question sets.

FrugalGPT reported that LLM API prices differed by up to two orders of magnitude at the time of study.
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance