Skip to content
ML Academy

project-048 · LLM and GenAI · banking

LoRA/QLoRA adaptation plan for fine-grained banking intents (implementation-ready, docs-only, GPU optional)

Project-045 compares prompting and a TF-IDF model on BANKING77. Would parameter-efficient fine-tuning of a small open-weight model close the remaining gap on confusable intents at a serving cost the bank can carry? Before anyone rents a GPU, write a plan that a colleague could execute: what is trained, on which data, with what memory budget, against which baseline, and with which stopping and go/no-go rules.

Licence
CC BY 4.0 for the dataset; base-model licence to be checked per candidate before download
Reference run
none

At a glance

  • Data

    BANKING77 (same files and splits as project-045) (public) · Train 10,003 and test 3,080 queries, 77 intents (download-verified)

    Details on data

  • Baseline

    The best arm from project-045 (TF-IDF + logistic regression, or the prompted LLM if it ran), on the identical evaluation sample.

    Details on baseline

  • Validation

    Same validation carve-out for early stopping and hyperparameters; one final evaluation on the shared test sample;

    Details on validation

  • Metric

    Macro-F1, accuracy, per-intent F1, calibration error if thresholds are used, peak GPU memory, training time, adapter size, inference latency per query.

    Details on metric

Success criterion

The plan is complete when a reviewer can run it without asking questions: data splits and deduplication rules are fixed, the base-model selection criteria (including licence checks) are explicit, the memory estimate is shown with its assumptions, the evaluation reuses project-045's split and metrics, and a go/no-go rule is stated (for example: adopt only if macro-F1 beats the best project-045 arm with an interval excluding zero at a lower cost per 1,000 queries under labelled hypothetical prices). Execution is optional and, if done, reported separately.

Context

Day 43 asks whether the missing capability calls for prompting, retrieval or fine-tuning; Day 44 covers LoRA, QLoRA and memory trade-offs; Day 45 covers evaluation. Fine-grained intent classification with a fixed label set and thousands of labelled examples is a case where adaptation can be justified, because the needed capability is a label boundary, not knowledge. This record is docs-only: no training run is required or claimed; a GPU makes the optional execution practical.

Data

Dataset
BANKING77 (same files and splits as project-045) (public)
Licence
CC BY 4.0 for the dataset; base-model licence to be checked per candidate before download
Size
Train 10,003 and test 3,080 queries, 77 intents (download-verified)
Verification
download verified

Access

GitHub CSVs (no login), hashed; reuse project-045's validation carve-out and stratified 770-query evaluation sample so results are comparable. Base-model weights are downloaded only if the plan is executed; check each candidate model's licence and acceptable-use terms at that time.

Grain

One training example = (query, intent) formatted as an instruction-style input with the label as the target.

ColumnMeaning
textCustomer query
categoryIntent label, the generation or classification target
splittrain (minus validation carve-out), validation, test sample shared with project-045
dedup_groupDerived: near-duplicate group id; groups never span train and evaluation

EDA and data-quality rules

  • Near-duplicate grouping (normalised text and high TF-IDF cosine) across train/validation/test; any group spanning splits is moved entirely into train or removed from evaluation, and the count is reported.

  • Token-length distribution of formatted examples with the candidate tokenizer to set max sequence length and estimate tokens per epoch.

  • Label balance per split; no intent with fewer than a minimum number of validation examples.

Baseline

The best arm from project-045 (TF-IDF + logistic regression, or the prompted LLM if it ran), on the identical evaluation sample. A second internal baseline: the same small base model with a linear classification head trained on frozen features, which shows how much LoRA adds over no adaptation of the backbone.

Features

Two formulations to choose between in the plan: (a) sequence classification head plus LoRA on attention projections; (b) instruction-style generation of the label string with constrained decoding. Plan (a) by default, because it cannot produce invalid labels. LoRA settings to sweep minimally: rank 8 and 16, alpha twice the rank, dropout 0.05, learning rate from a short range, 2-3 epochs with early stopping on validation macro-F1. QLoRA (4-bit base) as the memory fallback.

Candidate methods

  • LoRA on a small open-weight encoder or decoder with a classification head

    Few trainable parameters, adapters small enough to version per label-set release; outputs always valid labels.

  • QLoRA on a larger base

    Fits a stronger base into limited GPU memory; adds quantisation effects to measure and slower training.

Validation design

Same validation carve-out for early stopping and hyperparameters; one final evaluation on the shared test sample; three seeds for the chosen configuration to show run-to-run variance; paired bootstrap against project-045's best arm. Calibration check of the classifier probabilities if an abstain threshold is planned.

Leakage checks

Near-duplicate groups must not span splits; hyperparameters are never chosen on the test sample. If the base model's pretraining data may include BANKING77 (public since 2020), state it and add a small private probe set of learner-written queries.

Bias and representativeness

English-only, single-product queries; adapters trained here inherit the label taxonomy and phrasing. Report per-intent F1 and check that gains are not limited to the most frequent intents.

Robustness checks

Evaluate on paraphrase/typo perturbations and an out-of-scope probe set; fine-tuned classifiers may become more confident on out-of-scope inputs, which matters for routing.

Technical metrics

Macro-F1, accuracy, per-intent F1, calibration error if thresholds are used, peak GPU memory, training time, adapter size, inference latency per query.

Business interpretation

Misroutes per 1,000 messages vs project-045, one-off training cost and monthly serving cost under labelled hypothetical GPU and hosting prices, and the maintenance cost of retraining when intents change.

Error slices

  • Confusable intent pairs identified in project-045.

  • Short and out-of-scope queries.

  • Intents with the fewest training examples.

Implementation steps

  • Freeze splits and dedup groups shared with project-045; write the data card.

  • Write base-model selection criteria: licence permits the use, size fits the memory budget, tokenizer handles the domain, and a model card exists; shortlist without downloading.

  • Write the memory estimate: weights (parameters x bytes per parameter at the chosen precision) + LoRA parameters and their optimizer states + activations for the chosen batch and sequence length; state each assumption.

  • Write the training config (LoRA targets, rank, alpha, dropout, learning rate, epochs, early stopping, seeds) and the evaluation script interface.

  • Write the go/no-go rule, the cost model with hypothetical prices, and the rollback plan (keep the project-045 model serving).

  • Optional execution on a GPU: run the sweep, record peak memory and time, and report results as a separate, dated addendum.

Reproduction

Docs-only by default: the deliverable is the plan, the data card, the frozen splits and the config files. If executed, expected commands would look like python train_lora.py --config configs/lora_r8.yaml --seed 1 in an environment with PyTorch, Transformers and PEFT pinned to recorded versions; this requires model downloads and a GPU and is outside the academy's CPU-only reference set.

Limitations

  • Unexecuted plans can hide memory and time surprises; the estimate is an estimate until measured.

  • Gains on a public benchmark may not transfer to the bank's real intent taxonomy.

  • Adapters must be retrained whenever the label set changes, which is a recurring cost the classical baseline also has but at much lower expense.

Production and monitoring

Version base model, adapter, tokenizer and label set together; serve with a fallback to the project-045 model on errors or invalid outputs; monitor intent distribution and confidence; schedule retraining when new intents are added; keep the evaluation sample as a release gate.

Interview questions

  • Why fine-tune for this task rather than add retrieval or improve the prompt? What evidence would change your mind?

  • Walk through a memory estimate for LoRA versus full fine-tuning on the same base model.

  • How would you tell whether a fine-tuned classifier's gain is real or the result of near-duplicates between train and test?

Postmortem and alternatives

If the plan's go/no-go rule is likely to fail because the TF-IDF baseline is already strong, say so: the best outcome of a plan can be not training. Discuss data improvements (relabelling confusable pairs) as the cheaper alternative.

Rubric3 criteria

CriterionMeetsExceeds
ExecutabilitySplits, configs, memory estimate and evaluation interface are complete enough for another person to run.Includes a dry-run checklist and failure handling (out-of-memory, divergence) with fallbacks.
Decision logicGo/no-go rule tied to project-045's baseline and labelled hypothetical costs.Argues explicitly when not to train, with the evidence that would change the decision.
Evaluation integrityDedup groups, shared test sample, seeds and paired comparison specified.Out-of-scope and contamination probes planned with interpretation rules.

References6