LLMs, GenAI, RAG, agents, evaluation · insight-0553

Choose models on a task-specific eval set, not on public leaderboard rank

Situation
A team picks the model at the top of a public leaderboard for a booking-change assistant.
Action
Build an eval set from anonymised production-like cases with expected outputs and slices (route type, language, change type), score candidate models on quality, latency and cost per case, and use leaderboards only to shortlist.
Reason
Holistic evaluation work shows model rankings vary across scenarios and metrics such as accuracy, calibration, robustness and efficiency, so one aggregate number hides trade-offs that matter for a specific task.
Boundary
Model selection and vendor comparisons for any LLM feature.

Symptom or risk

The leader is slower, worse at the airline's own formats and no better on the real cases, and the choice is hard to reverse later.

Very early exploration may use public benchmarks before any data exists, as a temporary step.

Building the eval set takes days; it then serves every later decision.

Writing eval items that resemble public benchmark questions rather than your traffic.

Liang et al. (HELM) evaluate models across many scenarios and metrics, including accuracy, calibration, robustness, fairness and efficiency, arguing against reliance on single-number leaderboards.
Holistic Evaluation of Language Models