project-045 · LLM and GenAI · banking

Intent classification for banking support: LLM zero/few-shot vs a TF-IDF baseline on cost, accuracy and failure modes

A bank's support team routes incoming customer messages to 77 fine-grained intents. Is an LLM prompted with the label list (zero- or few-shot) better than a TF-IDF + logistic regression model trained on labelled examples, once accuracy, invalid outputs, cost per 1,000 messages and latency are counted? And how many labelled examples per intent does the classical model need before the LLM's advantage disappears?

Dataset: BANKING77 (online-banking queries, 77 intents)
Licence: CC BY 4.0 (GitHub repository licence and dataset card)
Reference run: none
Lessons: day 41 day 42 day 54

Data
BANKING77 (online-banking queries, 77 intents) (public) · Train 10,003 and test 3,080 queries, 77 intents in each (download-verified row counts)
Details on data
Baseline
TF-IDF (word 1-2 grams, sublinear tf) + multinomial logistic regression, fitted on train only, regularisation chosen on the validation carve-out.
Details on baseline
Validation
Classical models: stratified validation carve-out from train for tuning, then one fit on full train and one score on test.
Details on validation
Metric
Accuracy, macro-F1, per-intent F1, invalid-label rate, confusion between confusable pairs, cost per 1,000 queries from token usage, p50/p95 latency.
Details on metric

Success criterion

Recommend the LLM only if its macro-F1 on the stratified test sample exceeds the full-data TF-IDF model's with a bootstrap interval excluding zero, its invalid-label rate is below 1% after output validation, and its cost per 1,000 messages (hypothetical prices) is acceptable to the business owner. Otherwise recommend the classical model and state the label budget at which it matches the LLM.

Context

Day 41 compares TF-IDF with embedding approaches; Day 42 explains why fluent output may still be wrong; Day 54 prices the decision. BANKING77 is hard because intents are fine-grained and overlap (several card and transfer intents differ by one detail), which is exactly where label names alone may be insufficient for a prompted model.

Data

Dataset: BANKING77 (online-banking queries, 77 intents) (public)
Licence: CC BY 4.0 (GitHub repository licence and dataset card)
Size: Train 10,003 and test 3,080 queries, 77 intents in each (download-verified row counts)
Verification: download verified

Access

CSV files from the PolyAI-LDN/task-specific-datasets GitHub repository (banking_data/train.csv, test.csv; no login). The Hugging Face entry relies on a loading script that recent datasets versions do not run, so use the CSVs. Record file hashes.

Grain

One row per customer query with one intent label.

Column	Meaning
text	Customer query, usually one sentence
category	Intent label such as card_arrival or declined_transfer (77 values)
split	Original train/test split; a validation set is carved from train, stratified by intent
prediction / valid_label	Derived: model output and whether it maps exactly to one of the 77 labels

EDA and data-quality rules

Class balance in train and test; near-duplicate check (normalised text) between train and test, reporting the count before any modelling.
Label-confusability map: pairs of intents with high TF-IDF centroid similarity, used later to read the confusion matrix.
Length distribution; flag very short queries that are ambiguous without context.

Baseline

TF-IDF (word 1-2 grams, sublinear tf) + multinomial logistic regression, fitted on train only, regularisation chosen on the validation carve-out. Also a learning curve: the same model trained on 5, 10, 25 and 50 examples per intent and on the full train split, so the LLM can be placed against a labelling budget.

Features

Classical arm: vectoriser fitted inside the training split only. LLM arms: the 77 label names with one-line descriptions written by the learner, zero-shot and few-shot (k examples per intent drawn from train with a fixed seed, or a retrieved-nearest-examples variant), output constrained to the label list and validated; any output not in the list counts as an error after one deterministic normalisation (case, whitespace).

Candidate methods

Zero-shot LLM with label names and short descriptions
No labelled data needed; tests how much the label wording carries.
Few-shot LLM with retrieved nearest training examples
Gives the model in-domain examples for confusable intents; longer prompts raise cost.
Embedding + logistic regression (optional, needs a sentence-embedding model download)
Middle option between TF-IDF and an LLM; cheap at inference and often strong on intent data.

Validation design

Classical models: stratified validation carve-out from train for tuning, then one fit on full train and one score on test. LLM arms: to control cost, score a stratified test sample of 770 queries (10 per intent, fixed seed); the classical models are scored on the same 770 for the paired comparison and on the full test for reference. Bootstrap over queries for macro-F1 differences.

Leakage checks

Few-shot examples come from train only; the retrieved-examples variant must never retrieve from test. BANKING77 has been public since 2020, so the LLM may have seen it; record the model's stated training cut-off and treat its accuracy as possibly optimistic.

Bias and representativeness

Queries are English and written for one banking product; intent names follow that product. Report per-intent F1 and check whether errors concentrate in intents with longer or rarer phrasing.

Robustness checks

Paraphrase and typo perturbations on 50 queries; an out-of-scope probe set (queries from another domain written by the learner) to see whether each arm forces an in-scope label.

Technical metrics

Accuracy, macro-F1, per-intent F1, invalid-label rate, confusion between confusable pairs, cost per 1,000 queries from token usage, p50/p95 latency.

Business interpretation

Misrouted messages per 1,000 (each costs an agent transfer), labelling effort saved or required, and running cost under a labelled hypothetical price table.

Error slices

Confusable intent pairs (card vs transfer families).
Short queries (five words or fewer).
Out-of-scope probes: forced in-scope predictions.

Implementation steps

Download CSVs, hash them, run the EDA rules including the train-test near-duplicate check.
Fit the TF-IDF baseline and the learning curve; save predictions and per-intent metrics.
With a configured model: implement zero-shot and few-shot arms with validated outputs; log model id, date, prompt version and usage.
Score all arms on the stratified sample with paired bootstrap; read confusion matrices against the confusability map.
Write the recommendation including the label budget where TF-IDF matches the LLM.

Reproduction

Learner-authored; for example uv run python -m intents.baseline --seed 5 (CPU only, minutes) and, with a model configured, uv run python -m intents.llm --mode fewshot --k 3 --sample 770. Without a model, the deliverable is the classical baseline, the learning curve and the evaluation harness, and the report states that the LLM comparison was not run.

Limitations

A 770-query sample gives about ten queries per intent, so per-intent F1 is noisy.
Possible contamination makes the LLM look better than it would on a private intent set.
Hypothetical prices: the cost conclusion depends on them.

Production and monitoring

Whatever wins, add an abstain or out-of-scope path and monitor the label distribution weekly. For an LLM arm, validate every output against the label list and fall back to the classical model on invalid output. Retrain or re-prompt when new intents appear, and re-run this comparison when the label set changes.

Interview questions

Why might a 30-line TF-IDF model beat a large model on a 77-way intent task?
How do you make an LLM classifier's output safe to route on?
What would convince you the LLM's accuracy is not memorisation of a public dataset?

Postmortem and alternatives

If the classical model wins with full data but the LLM wins at five examples per intent, the decision is about the labelling budget and launch time, not about models. Discuss a hybrid: LLM labels to bootstrap, classical model to serve.

Rubric3 criteria

Criterion	Meets	Exceeds
Baseline strength	Tuned TF-IDF model and a learning curve on the same splits.	Confusability analysis explains where each arm fails.
LLM arm honesty	Validated outputs, logged model version, cost from usage, or explicitly not run.	Out-of-scope and contamination checks reported with their limits.
Decision	Recommendation by the pre-set rule with paired intervals.	States the label budget break-even and a production fallback design.

References5

Efficient Intent Detection with Dual Sentence Encoders
Dataset introduction and few-shot intent detection context.
sklearn.feature_extraction.text.TfidfVectorizer
Baseline vectoriser and fit-on-train discipline.
Language Models are Few-Shot Learners
Few-shot in-context learning as the LLM arm's mechanism.
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Sentence embeddings for the optional middle arm.
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Public benchmark contamination risk for the LLM arm.

At a glance