LLMs, GenAI, RAG, agents, evaluation · insight-0507

For fixed-label text classification, benchmark a small encoder before a generative LLM

Situation
A team wants a large generative model to tag two million complaint emails a month into 12 fixed categories.
Action
On the same labelled split, compare TF-IDF with logistic regression, a fine-tuned encoder (BERT-class) and the prompted LLM; decide on quality weighted by error cost, latency and volume.
Reason
Encoders pretrained bidirectionally and fine-tuned with a small classification head were built for this kind of task and are cheap to serve; a generative model's flexibility adds little for a fixed label set.
Boundary
Stable label sets with at least a few thousand labelled examples.

Symptom or risk

The pipeline is slow, hard to calibrate and expensive at volume, and nobody knows whether a smaller model would match it.

Few labels, labels that change often, or tasks that also need extraction or explanations favour prompting an LLM.

Fine-tuning needs labels and a deployment path; prompting starts faster.

Testing the LLM on curated examples and the encoder on the full messy test set.

Devlin et al. showed a pretrained BERT encoder fine-tuned with one additional output layer reached strong results across tasks including sentence classification.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding