Core day 41 of 60 · Week 9 · Part II · application depth

NLP Foundations

The support desk has 60 labelled complaint messages and 600 unlabelled ones from the same inbox, and a vendor offers an "AI language model" to route them. What is the strongest simple text baseline, what would a dense representation have to add to beat it, and how do you split the data so that near-duplicate messages cannot flatter the result?

By the end: For a small text-classification task you fit a TF-IDF plus logistic regression baseline, compare it with an LSA (SVD) representation on familiar and unfamiliar phrasing, and show with a grouped split how near-duplicate messages inflate an accuracy number.

At a glance

Turn text into weighted word counts, fit a linear model as the baseline, and only then ask whether a dense representation adds anything.

Tokens to columns
Split text into tokens; every vocabulary word becomes one sparse column.
TF-IDF
Count times rarity: words in every message fade, specific words dominate.
Linear baseline
Logistic regression on TF-IDF is cheap, fast and readable: run it first.
Vocabulary mismatch
Unseen words get no weight; LSA or encoders help only if a corpus links them.
Grouped split
Split by ticket or template; near-duplicates across a random split inflate scores.

Recall · 5 min

No reviews due? Answer from memory. (1) Why must a scaler or encoder be fit on the training rows only (Day 9)? (2) What did a strong simple baseline have to beat on Day 1? (3) In your own words, what does an embedding give you that a one-hot column does not (Day 38)? Today's vectorizer is a preprocessing step with the same boundary rule.

Open due reviews

Concept · 10 min

A model cannot read, so a message must become numbers. Counting words (Bag-of-words), weighting rare ones up (TF-IDF) and fitting a linear classifier gives a strong, cheap, readable baseline. It fails where the words change (Vocabulary mismatch); a dense representation helps only if its corpus links the old and new words.

A TF-IDF baseline fails exactly where the words change, and a dense representation helps only if the corpus links the words

Predict first

A query says "suitcase" but the only document that answers it says "luggage". Which method can still return that document?

Text description of the figure

A picker of 40 labelled queries (28 keyword, 12 paraphrase), or a typed query, over 567 documents from the academy's own glossary, insights and lesson text. Controls set k, the number of LSA dimensions (the dense representation here is LSA, not a neural model) and whether to show TF-IDF, LSA or both. Each method lists its top documents with cosine scores, the words shared with the query and whether each document is labelled relevant, then reports recall, precision, reciprocal rank and nDCG for the query and the mean over all labelled queries, with a bootstrap interval for the gap.

Four ideas to hold on to Overview

Count
A message becomes a row of word counts; order is thrown away.
Weight
TF-IDF damps words found everywhere and lifts rare, specific ones.
Densify
SVD or an encoder places co-occurring words near each other.
Split by group
Copies of one ticket on both sides of a split measure memory.

From message to intent Diagram

Messageraw text
Tokenslowercase, split, n-grams
Count matrixone column per word, sparse
- to TF-IDF weights
Unlabelled poolno labels needed
- fit on: to SVD, k dimensions
TF-IDF weightscount times rarity
- to Logistic regression
project
SVD, k dimensionsLSA: co-occurring words merge
optional
Logistic regressionreadable word weights

Every fitted step (vocabulary, IDF, SVD) is preprocessing and obeys the Day 9 boundary. The SVD reads no labels, so the unlabelled pool is free data for it.

A paraphrase is invisible to TF-IDF and obvious to LSA Illustration

TF-IDF scores a message only through shared words. No labelled baggage message contains carousel or belt, so they share only filler words with the query and sit among the other intents. Baggage in its top 5: 1.

LSA, fitted on the 600 unlabelled messages where carousel and belt co-occur with luggage, lifts all 20 baggage messages to the top without a single shared topic word. Baggage in its top 5: 5.

Synthetic inbox (seeded): query “the carousel belt again please”; 60 labelled messages (filled dots: baggage) written only with the older words; IDF and LSA fitted on the 600 unlabelled plus 60 labelled messages with the academy’s retrieval code.

One query in the newer words against 60 labelled messages written in the older ones. Same data, two representations; only LSA saw the co-occurrence.

Three text representations Comparison

Criterion	TF-IDFlexical baseline	LSAtruncated SVD of TF-IDF	Neural encoderpretrained, docs-only here
What a message becomes	Sparse row, one weight per vocabulary word	k dense numbers per message	One dense vector per message from a pretrained model
Links two synonyms	No: different words are unrelated columns	Yes, if they co-occur in the fitting corpus	Often, from the encoder's own training corpus
Needs	Labels that cover the phrasing	An unlabelled pool from the same inbox	A model download, a version to pin, latency
Inspect and operate	Readable weights; milliseconds; nothing to download	Dimensions are blends of words; choose k on validation	Opaque; re-embed everything when the model changes

Run the left column first. Each step to the right must win on the same grouped split to earn its cost.

Which representation the situation calls for Map

TF-IDF is enough

Few labels, but the words repeat: the baseline already separates intents.

TF-IDF is hard to beat

Many labels covering the words: dense models rarely add much.

Dense helps most

Few labels, new words: LSA or an encoder can transfer weight.

Label the new words

Many labels, but drift: label recent messages, then compare.

A practitioner rule of thumb, to test on your own grouped split.

Same model, same messages: the split alone moves accuracy from 0.56 to 0.98 Chart

Random split of messages87% of test messages have a near-copy in training0.98 accuracy
Split grouped by ticket0% near-copies: the number production will see0.56 accuracy

Optional lab: 150 synthetic tickets, each resubmitted three times with small edits. Near-copy = TF-IDF cosine 0.8 or more to a training message. In the random split, 100 of the 104 test tickets also sit in training.

Pipeline: Tokenization -> counts -> TF-IDF weights -> logistic regression (Day 11).
Dense option: Latent semantic analysis (LSA) (truncated SVD of the TF-IDF table) or a neural encoder (Day 38).
Evaluation: split by ticket or template so near-duplicates cannot flatter the score.
Details are in the pictures above and the optional technical depth below.

Assumptions

Word presence is informative enough that discarding order and negation ("not delayed") costs little; routing usually passes, clause-level sentiment may not.
Labelled and future messages share most of their vocabulary; when they do not, dense representations or more labels are needed.
Near-duplicates are identifiable by a group key or a similarity check, so the split can respect them.

Application and workplace decision · 10 min

Data (synthetic): 60 labelled complaints, 600 unlabelled, four intents (baggage, delay, refund, seat).
Labelled messages use three phrasings per intent; this year's inbox adds two newer ones, which co-occur with the old ones in the unlabelled pool.
The code fits TF-IDF and an 8-dimensional LSA, trains one logistic regression on each and scores familiar and new phrasing; a second pass trains on labels that already include the new words.
Predict first: how does TF-IDF score on new phrasing?

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(41)
topics = [  # synthetic complaint intents; the last two words of each are newer phrasings
    "luggage suitcase bag carousel belt".split(),
    "delayed late postponed waiting rebooked".split(),
    "refund reimbursement voucher compensation chargeback".split(),
    "seat upgrade legroom aisle window".split(),
]
filler = "the my on flight again very still please staff gate told after".split()

def make(n, lo, hi):  # topic words drawn from positions lo..hi of each intent
    y = rng.integers(0, 4, n)
    docs = [" ".join(rng.choice(topics[i][lo:hi], 2).tolist() + rng.choice(filler, 5).tolist()) for i in y]
    return docs, y

pool, _ = make(600, 0, 5)  # unlabelled history: all phrasings, no labels
old_x, old_y = make(300, 0, 3)  # test: familiar phrasing
new_x, new_y = make(300, 3, 5)  # test: phrasing absent from the labelled set
for vocab in (3, 5):  # labelled set uses only familiar words, or all words
    tr_x, tr_y = make(60, 0, vocab)
    tfidf = TfidfVectorizer().fit(pool + tr_x)
    svd = TruncatedSVD(8, random_state=0).fit(tfidf.transform(pool))
    for name, f in {"tfidf": tfidf.transform, "lsa": lambda d: svd.transform(tfidf.transform(d))}.items():
        clf = LogisticRegression(C=10, max_iter=1000).fit(f(tr_x), tr_y)
        print(f"train words<={vocab} {name:<5} familiar={clf.score(f(old_x), old_y):.2f} new={clf.score(f(new_x), new_y):.2f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 27 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

TF-IDF, derived and computed by hand4 minThe exact weighting scikit-learn uses, pushed through a three-message corpus on paper.

TF-IDF as scikit-learn computes it (smooth_idf, the default) Formula

w_{t,d} = \mathrm{tf}_{t,d} \cdot \left(\ln\frac{1+N}{1+\mathrm{df}_t} + 1\right)

$\mathrm{tf}_{t,d}$: count of term t in message d
$N$: number of messages the vectorizer was fitted on
$\mathrm{df}_t$: number of those messages that contain t
$+1$: added after the log, so a word in every message keeps weight 1; the 1+ inside the log is the smoothing

Each row is then scaled to unit length (L2), so cosine similarity is a plain dot product.

IDF weights on four tiny messages (synthetic card-support inbox) Chart

card (3 of 4 messages)1.223 idf
payment (3 of 4)1.223 idf
my (2 of 4)1.511 idf
refund (1 of 4)1.916 idf
loan (1 of 4)1.916 idf

Messages: my card payment failed; card charged twice for payment; refund my card fee; loan payment due date. Computed with TfidfVectorizer defaults.

TF-IDF by hand: d1 of three returns-desk messages (e-commerce, synthetic) Table

Term	df of 3 docs	idf = ln(4/(1+df)) + 1	raw = tf x idf	after L2
refund	2 (d1, d2)	ln(4/3) + 1 = 1.288	1 x 1.288 = 1.288	1.288 / 3.008 = 0.43
for	1 (d1)	ln(4/2) + 1 = 1.693	1 x 1.693 = 1.693	1.693 / 3.008 = 0.56
late	1 (d1)	1.693	1.693	0.56
delivery	2 (d1, d3)	ln(4/3) + 1 = 1.288	1.288	0.43
L2 norm of d1	-	-	sqrt(2 x 1.288^2 + 2 x 1.693^2) = 3.008	divide each raw weight by it

d1 = "refund for late delivery", d2 = "damaged parcel refund", d3 = "update delivery address". Every number recomputed with TfidfVectorizer defaults (smooth_idf, L2): the fitted rows match to three decimals.

The fitted TF-IDF matrix: each message is a sparse unit row Grid

Vocabulary word →
Message ↓	refund	for	late	delivery	damaged	parcel	update	address
d1 refund for late...	0.43	0.56	0.56	0.43	0.00	0.00	0.00	0.00
d2 damaged parcel...	0.47	0.00	0.00	0.00	0.62	0.62	0.00	0.00
d3 update delivery...	0.00	0.00	0.00	0.47	0.00	0.00	0.62	0.62

All three rows from TfidfVectorizer defaults (computed); row d1 is the worked table above. Words in two messages (refund, delivery) weigh less than words in one. Most cells are 0: a message only has weight on its own words, which is why cosine needs overlap.

Fit on training text, transform test text: an unseen word simply vanishes Diagram

Training + pool textunlabelled pool included; no labels needed
- fit on: to fit: vocabulary + IDF
Test messagescored once, at the end
- transform only: to transform
never (leaks)
fit: vocabulary + IDFone column per word seen here
fixed columns
transformfitted columns only
- silently: to Unseen word
weighted
Known wordcount x fitted IDF
Unseen worddropped: no error, no weight

The Day 9 boundary applied to text. Refitting on test text would add test-only words to the vocabulary and IDF. The quiet failure is the other branch: the pipeline runs, and the new word contributes nothing to the row.

Defaults that matter (scikit-learn 1.9.1, checked in this repo) Table

Setting	Default	Why an expert changes it
TfidfVectorizer token_pattern	words of 2+ characters	Single letters and digits vanish, so a code written as A 7 loses its parts
ngram_range	(1, 1)	Bigrams keep short phrases and negation as one column
sublinear_tf	False	True uses 1 + log tf, so a repeated word does not dominate
min_df / max_df	1 / 1.0	Prune one-off typos and near-universal words on large corpora
TruncatedSVD n_components	2	Always set it; choose k on a grouped validation split
LogisticRegression C, max_iter	1.0, 100	Tune C; raise max_iter on wide sparse data if it warns

Verified with get_params() on the installed versions. The application code sets C = 10 and k = 8 explicitly.

Cosine retrieval, by hand4 minThe retrieval score as a dot product of unit rows, computed on the same three messages, and where lexical scoring wins or loses.

Cosine similarity, the retrieval score Formula

\cos(q, d) = \frac{q \cdot d}{\lVert q \rVert\,\lVert d \rVert}

$q \cdot d$: sum over shared vocabulary words of the two weights multiplied
$\lVert q \rVert$: L2 length; already 1 for TfidfVectorizer rows
$\cos = 0$: no shared word with nonzero weight

With L2-normalised rows the cosine is a plain dot product, so ranking all 600 messages against a query is one sparse matrix product, with no loop. That is why the vectorizer normalises by default.

Three queries against the three returns-desk messages: cosine needs a shared word Grid

Message →
Query ↓	d1 late delivery	d2 damaged parcel	d3 address change
late delivery refund	0.83	0.25	0.25
where is my parcel	0.00	0.62	0.00
where is my package	0.00	0.00	0.00

TfidfVectorizer fitted on the three messages (computed). Row 1: refund ties d2 and delivery ties d3. Row 2: one shared informative word is enough. Row 3: package was unseen at fit time, so it is dropped and every score is exactly 0, the vocabulary-mismatch failure the lesson turns on. A 0.00 is about shared words, not relevance.

Lexical versus dense, in retrieval Comparison

Criterion	Lexical (TF-IDF, BM25)	Dense (LSA, encoders)
Strong on	Identifiers, codes, rare exact terms	Paraphrase, synonyms present in its corpus
Weak on	Paraphrase with no shared words	Rare identifiers with no semantic neighbours
Typical use	Baseline; lexical half of hybrid search (Day 48)	Semantic half of hybrid search

A practitioner heuristic; test it on your own labelled queries.

BM25, the lexical scorer behind most search engines Formula

s(q,d) = \sum_{t \in q} \mathrm{idf}_t \cdot \frac{f_{t,d}\,(k_1+1)}{f_{t,d} + k_1\left(1 - b + b\,\frac{|d|}{\mathrm{avgdl}}\right)}

$f_{t,d}$: count of term t in document d
$k_1$: saturation: with k1 = 1.5 and b = 0, counts 1, 2, 3, 5, 10 weigh 1.0, 1.43, 1.67, 1.92, 2.17
$b$: length normalisation: 0 ignores length, 1 fully divides by relative length
$|d| / \mathrm{avgdl}$: document length relative to the corpus average

Robertson and Zaragoza 2009. The academy's retrieval code uses k1 = 1.5, b = 0.75; libraries differ, so record yours. Saturation numbers computed in Python.

LSA as a low-rank approximation, by handadvanced6 minThe best rank-k factorisation derived, pushed through five messages on paper, and why it lets a new word borrow the weight of an old one.

LSA is a rank-k approximation Formula

X \approx U_k \Sigma_k V_k^{\top}

$X$: TF-IDF matrix, messages by words
$U_k \Sigma_k$: k numbers per message (the dense representation)
$V_k$: k directions over words; co-occurring words load on the same direction

TruncatedSVD does not center the data, so it works on the sparse matrix directly; applied to a TF-IDF matrix, scikit-learn documents it as LSA. A randomized truncated SVD costs roughly nonzeros x k, so k = 100 on a million messages is routine; a full SVD is not.

Why truncation is the right cut: the best rank-k approximation Formula

\min_{\mathrm{rank}(Y) \le k} \lVert X - Y \rVert_F^2 = \lVert X - X_k \rVert_F^2 = \sum_{i > k} \sigma_i^2

$\sigma_1 \ge \sigma_2 \ge \dots$: singular values of X; their squares add up to the sum of all squared entries of X
$X_k = U_k \Sigma_k V_k^{\top}$: keep the k largest singular values and their vectors; drop the rest
$\sum_{i>k} \sigma_i^2$: the squared error you pay is exactly the dropped singular values squared (Eckart-Young theorem)
$x V_k$: fold-in: a new message x (a TF-IDF row) is placed in the space by multiplying it by the word directions

No other rank-k matrix reconstructs X better in squared error. Small k forces co-occurring words onto shared directions, which is the whole trick.

LSA by hand: five returns-desk messages, rank 2 (e-commerce, synthetic) Table

Quantity	Value	What to check on paper
Messages (word counts)	d1 parcel lost; d2 package lost; d3 lost parcel package; d4, d5 change address	X is 5 messages x 5 words, eleven 1s
Singular values	2.414, 2.000, 1.000, 0.414, 0	Squares add to 11, the squared entries of X
Direction 1 (v1)	parcel 0.50, package 0.50, lost 0.71	The delivery topic: parcel and package load equally
Direction 2 (v2)	address 0.71, change 0.71	The address topic, orthogonal to direction 1
Rank-2 error	1.000 + 0.172 = 1.172	Exactly the two dropped singular values squared
Row d1 of X2	parcel 0.60, package 0.60, lost 0.85	d1 never said package, yet now carries weight on it
cos(query package, doc parcel)	raw 0.00, rank 2: 1.00	Fold both in with V2: they land on direction 1
cos(query package, doc address)	raw 0.00, rank 2: 0.00	Words that never co-occur stay apart

Raw counts instead of TF-IDF keep the arithmetic hand-sized; checked with numpy.linalg.svd (signs of singular vectors are arbitrary). The 1.00 is the caveat in miniature: at rank 2 parcel and package become indistinguishable.

Why an unseen word inherits weight Sequence

Co-occurrence
In the pool, suitcase appears in messages that also say luggage.
Shared direction
Their columns of X are similar, so SVD maps them to the same directions.
Classifier
Logistic regression learns weights on directions, not on words.
Transfer
A test message with only suitcase lands where luggage messages did.
Caveat
Unrelated words that merely co-occur merge too: inspect nearest neighbours.

The number of LSA dimensions decides whether meaning transfers Illustration

Too few dimensions merge the intents. From k = 3 to 12 the new words land on their old synonyms (accuracy 1.00 first reached at k = 4, the number of intents). At k = 32 the space is the full word table again and LSA falls back to the TF-IDF level (0.32).

Share of the TF-IDF table's squared norm (energy) the k dimensions keep. At k = 4 only 35% is kept, yet routing is already perfect: the useful structure is a handful of co-occurrence directions, the rest is filler noise.

Synthetic inbox (seeded): 600 unlabelled and 60 labelled messages (older words only), 150 test messages in the newer words; 32 distinct words, so k = 32 is full rank. 1-nearest-neighbour routing stands in for the classifier of the application code.

Computed routing accuracy on new phrasing for every k, next to the share of the matrix the k dimensions keep. In practice pick k on a grouped validation split and report this whole curve, not one value; a flat region is the safe choice.

Properties and failure modes: what a bag of words cannot see3 minUnigrams, bigrams, character n-grams and stop lists each decide what the model can see, and each blind spot has a diagnostic.

Tokenization choices and what they change Table

Choice	Helps with	Costs or breaks
Word unigrams	Fast, readable baseline	No order: not delayed equals delayed
Word bigrams	Keeps short phrases such as not delayed	Many more, sparser columns
Character n-grams	Typos, inflections, codes such as ZX104	Large vocabulary; weights harder to read
Stop-word list	Removes filler words	scikit-learn's English list contains not, no, never
Subword (BPE)	Open vocabulary for neural models (Day 42)	Token counts differ by language

A stop list can delete the label: two loan messages become the same row Table

Vectorizer setting	Columns it keeps	Can the two rows differ?
stop_words="english"	approved, loan	No: identical rows, no model can separate them
No stop list (default)	approved, is, loan, not, your	Yes: not is its own column
No stop list, ngram_range=(1, 2)	the five words + is approved, is not, loan is, not approved, your loan	Yes, and not approved is one column

"your loan is approved" against "your loan is not approved" (bank, synthetic), fitted with TfidfVectorizer (computed). scikit-learn's English stop list contains not, no and never, so the one word that carries the label is removed before any weighting.

Three blind spots of a bag of words, each with its own diagnostic Comparison

Criterion	Vocabulary mismatch	Order and negation	Sparsity
What you see	Cosine exactly 0 between a query and its answer	"loan approved" and "loan not approved" differ by one column at most	Tens of thousands of columns for a few hundred labels
Diagnostic	Share of test messages whose words are mostly unseen at fit time	Read errors grouped by the presence of negation words	Gap between train and grouped validation scores
First fix	Labels that cover the new words, or LSA / an encoder	Phrase features (bigrams), then a sequence model	Regularisation strength C; min_df pruning

Logistic regression copes with sparsity through regularisation; it cannot recover a column the tokenizer never made.

Near-duplicate leakage3 minHow resubmitted tickets inflate scores and the split that prevents it.

What a random split of resubmitted tickets puts in the test set Illustration

Random split of messages: 98 of the 102 tickets in the test part also have a copy in training, and 69% of test messages have a near-copy there. A model can score well by remembering ticket codes.

Split by ticket: no ticket crosses the boundary and 0% of test messages have a near-copy. What remains is shared topic and filler words, the situation production will look like.

Synthetic tickets (seeded), 150 tickets x 3 edited copies, as in the optional lab. Similarity is TF-IDF cosine with IDF from all messages; only the split differs between the panels. A TypeScript re-implementation with its own seed, so its shares differ from the Python lab output.

Cosine from each test message to its closest training message, for the two splits of the same 450 messages.

Splitting text without duplicate leakage Diagram

Messagestickets, forwards, templates
Group key available?
- yes: to Split by group
no
Cluster near-copiescosine >= 0.8 forms a group
as group
Split by groupticket, customer or template
Check test-to-trainreport share of test with cosine above 0.8

A group key keeps every copy of a ticket on one side. Without one, near-copies are clustered first so that each cluster acts as the group.

Neural sentence encoders, fairly comparedadvanced2 minThe protocol for testing a pretrained encoder against the baseline without flattering it.

Trying a neural encoder fairly (docs-only) Sequence

Pin
Record model name, version, embedding size and maximum input length.
Embed once
Embed labelled and unlabelled messages; cache the vectors.
Same head
Train the same logistic regression on the vectors.
Same split
Compare with a tuned TF-IDF baseline on identical grouped folds, with intervals.
Price it
Add latency, cost and re-embedding work before deciding.

Diagnosing errors and the order of work3 minWhich error kind points to which fix, from group key to decision.

Reading the baseline's errors before choosing a fix Diagram

Baseline errorsgrouped split
Words unseen at fit?
- yes: to LSA or encoder
no
Order or negation?
- yes: to Phrase features first
no
Labels inconsistent?
- yes: to Relabel with a guide
LSA or encoderfit on unlabelled pool
- to Re-score same split
Phrase features firstthen a sequence model
- to Re-score same split
Relabel with a guideno model fixes noise
Re-score same splitwith an interval

A practitioner diagnostic order. Each branch names a different cause, and only the first is a representation problem.

The order of work, from group key to decision Sequence

Group key
Choose ticket, customer or template before anything is fitted.
Tuned baseline
TF-IDF plus logistic regression, with a small search over n-grams and C.
Read 20 errors
Tag each one: unseen words, order or negation, or label noise (the diagram above).
Dense candidate
LSA or an encoder on the same grouped split, with the same head and metric.
Price the decision
Intervals on the grouped score and cost per routed message decide; novelty does not.

Interview nuances and common misreadingsadvanced2 minWhat separates a staff-level answer about text baselines from a textbook one.

What a staff-level answer adds Overview

Split before model
Names the group key and the leakage check before any model is mentioned.
Errors
Reads the top weights and 20 errors to name the failure kind.
Corpus
Says a dense space only links words its corpus put together.
Total cost
Prices latency, re-embedding on model change and a pinned version.
Misreading
LSA dimensions are not named topics; they are blends of words that change with k.

Memory anchor

Baseline first on a grouped split; a dense representation earns its place only where the words differ.

Useful for

Setting a text baseline in an afternoon, with readable weights, before any language model is considered (an e-commerce returns inbox, a bank's dispute notes).
Routing, tagging and deduplicating messages (insurance claim descriptions, telecom fault reports), or building the lexical half of a hybrid search (Day 48).
Diagnosing whether a failure is vocabulary mismatch (a representation problem) or label scarcity (a data problem).

Not suitable for

Tasks where word order or negation decides the label (a credit-card dispute that is "not authorised" versus "authorised", contradiction in clinical notes); a bag of words cannot separate them.
Treating LSA, or any embedding, as proof of understanding; similarity in the space is a corpus statistic, not an explanation.

Traps

Scoring on a random split of duplicated text

Resubmitted tickets, forwarded threads and template replies (an insurer's claim correspondence is full of them) put copies of one message on both sides of the split. The model is rewarded for remembering ticket-specific words, so offline accuracy is high and production accuracy is not.

How to avoid it

Split by a group key (ticket, customer, template) before fitting anything, report the nearest-neighbour similarity between test and train, and deduplicate near-copies when the group key is unavailable.

Comparing a tuned dense model with an untuned lexical baseline

In a product-review sentiment study the baseline is run with default settings and a stray stop-word list, the encoder gets a tuned head, and the encoder "wins". The difference is effort, not representation.

How to avoid it

Give the baseline the same split, a small search over n-grams and regularisation, and the same metric, then compare on identical grouped folds.

Senior reasoning

A senior treats the representation as a hypothesis about what separates the classes, whether the text is a telecom fault report, a patient-portal message or a bank's dispute note. They run the lexical baseline first, read its largest weights, and look at the errors: if they are unseen phrasings, a dense representation or more labels is justified; if they are label noise or duplicates, a bigger model will not help. They report the baseline and the candidate on the same grouped split (by ticket, patient or account), with intervals, and they price the extra model's latency, versioning and failure modes before adopting it.

Interview question

A clinic's patient portal receives free-text messages, and your team proposes a sentence-embedding model to route them into 12 triage categories. How do you decide whether it is worth it over a lexical baseline?

Outline your answer first.

Recall cards scheduled after completion

What does TF-IDF up-weight and down-weight, and what can it not do?
Why can a random split of customer messages overstate accuracy, and what replaces it?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

sklearn.feature_extraction.text.TfidfVectorizer — scikit-learn developerscore
Defines the smoothed IDF and the L2 row normalisation that today's baseline uses; the mechanism step 3 matches its description.
A statistical interpretation of term specificity and its application in retrieval — Karen Sparck Jones, 1972optional
Origin of weighting terms by their rarity across documents (the idea behind the IDF factor).
Decomposing signals in components (matrix factorization problems), section 2.5.3 Truncated singular value decomposition and latent semantic analysis (scikit-learn User Guide) — scikit-learn developerscore
Describes TruncatedSVD as LSA on tf-idf matrices and notes that it does not require centering, as used in the application block. (Section 2.5.3, Truncated singular value decomposition and latent semantic analysis)
Indexing by latent semantic analysis — Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, Richard Harshman, 1990optional
The original proposal to factorise a term-document matrix so that related words share dimensions.
Bag of Tricks for Efficient Text Classification — Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov, 2017optional
Evidence that simple linear text classifiers can match deep classifiers in accuracy on the tasks studied, which is why the lexical baseline comes first.
Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation — Aparna Elangovan, Jiayuan He, Karin Verspoor, 2021core
Shows that train-test overlap inflates NLP results by rewarding memorisation, the mechanism behind the grouped split.
Deduplicating Training Data Makes Language Models Better — Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini, 2022optional
Documents near-duplicates in language-model datasets, including overlap between training and validation text.
Speech and Language Processing (3rd edition draft) — Daniel Jurafsky, James H. Martin, 2026optional
Standard chapters on tokenization, TF-IDF weighting, vector semantics and text classification. (Chapters on vector semantics and embeddings, and on logistic regression for text classification)
The Probabilistic Relevance Framework: BM25 and Beyond — Stephen Robertson, Hugo Zaragoza, 2009optional
BM25, the lexical scorer that refines TF-IDF in search engines; named in the deep dive.
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Nils Reimers, Iryna Gurevych, 2019optional
Neural sentence embeddings (named here, docs-only) that replace steps 1-5 with a single encoder call.

Optional sections

Deep dive · 10 min

Why SVD finds synonyms. Two words that occur in the same messages have similar columns in $X$ . The best rank- $k$ approximation keeps the directions that explain most of the variation, and similar columns project onto the same directions, so a message containing "suitcase" and one containing "luggage" end up close in the $k$ -dimensional space even if they share no word. The same mechanism merges unrelated words that merely co-occur (a product brand and a seasonal discount code), which is why you inspect the nearest neighbours of a few terms before trusting the space.

Choosing $k$ . Too small and distinct intents collapse; too large and the space reproduces the sparse table. In the application code, $k=8$ for four intents leaves room for filler structure. In practice, pick $k$ by validation on a grouped split and report the sensitivity.

Lexical versus dense in retrieval. For queries that contain exact identifiers (SKU numbers, error codes, a policy clause number), a lexical method is often better than a dense one, because identifiers are rare tokens with high IDF and no semantic neighbours (a practitioner heuristic, to be tested on your queries). Dense methods help for paraphrase. Day 48 combines both. BM25 (Robertson and Zaragoza, 2009) is a lexical scoring function widely used in search engines as a refinement of plain TF-IDF.

Neural embeddings (docs-only). A pretrained sentence encoder returns one vector per message. Typical use: embed the 60 labelled and all unlabelled messages once, train logistic regression on the vectors, and compare with the TF-IDF baseline on the same grouped split. Pin the model name and version, record the embedding dimension and the maximum input length, and re-embed when any of them changes. The course runs no model downloads; the pattern is described in the Sentence-BERT paper and the SentenceTransformers documentation.

Retrieval project link (weekend studio). Saturday's retrieval milestone applies today's baseline to the academy's own text: TF-IDF vectors with cosine similarity over chunks of lessons, glossary and insights. Chunk ids that keep the source record in the id let you split or score by record, which is the grouped-split idea applied to retrieval, and the printed top-1 score on unanswerable queries is the lexical analogue of the nearest-neighbour check.

Hands-on lab: Measure the damage a random split does with near-duplicate tickets · 15 min

Run the code and read the three numbers: accuracy on a random split, accuracy on a split grouped by ticket, and the share of test messages that have a training neighbour above 0.8 cosine similarity. Then change the probability of dropping a word from 0.1 to 0.4 and rerun: how does the gap move, and why? Finally, add a near-duplicate filter that removes training messages above 0.8 similarity to any test message, and compare it with the grouped split.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupShuffleSplit, train_test_split

rng = np.random.default_rng(7)
topics = ["luggage suitcase bag".split(), "delayed late waiting".split(),
          "refund voucher compensation".split(), "seat upgrade legroom".split()]
filler = "the my on flight again very still please staff gate told after".split()

# 150 synthetic tickets; each is re-submitted three times with small edits (near-duplicates)
docs, labels, groups = [], [], []
for t in range(150):
    y = int(rng.integers(0, 4))
    topic_word = rng.choice(topics[y] if rng.random() < 0.6 else topics[int(rng.integers(0, 4))])
    base = [topic_word] + rng.choice(filler, 4).tolist() + [f"zx{rng.integers(100, 999)}", f"pnr{rng.integers(1000, 9999)}"]
    for _ in range(3):
        words = [w for w in base if rng.random() > 0.1]  # copy with ~10% of words dropped
        if rng.random() < 0.5:
            words.append("please")
        docs.append(" ".join(words))
        labels.append(y)
        groups.append(t)
docs, labels, groups = np.array(docs), np.array(labels), np.array(groups)

def evaluate(train_idx, test_idx, name):
    vec = TfidfVectorizer().fit(docs[train_idx])
    xtr, xte = vec.transform(docs[train_idx]), vec.transform(docs[test_idx])
    acc = LogisticRegression(C=10, max_iter=1000).fit(xtr, labels[train_idx]).score(xte, labels[test_idx])
    nn = (xte @ xtr.T).max(axis=1).toarray().ravel()  # cosine to nearest training message (rows are L2-normalised)
    print(f"{name:<14} accuracy={acc:.2f}  test msgs with a train neighbour >= 0.8: {(nn >= 0.8).mean():.0%}")

idx = np.arange(len(docs))
r_tr, r_te = train_test_split(idx, test_size=0.3, random_state=0)
evaluate(r_tr, r_te, "random split")
g_tr, g_te = next(GroupShuffleSplit(1, test_size=0.3, random_state=0).split(idx, groups=groups))
evaluate(g_tr, g_te, "grouped split")
print("shared tickets across random split:", len(set(groups[r_tr]) & set(groups[r_te])), "of", len(set(groups[r_te])))
print("shared tickets across grouped split:", len(set(groups[g_tr]) & set(groups[g_te])))

Interview practice · 10 min

A colleague shows 0.97 accuracy for a TF-IDF classifier that routes an insurer's claim emails and wants to ship it. The data contain forwarded email threads. What do you ask for before you agree?

Answer as you would in the interview.

Content version 7 · last verified 2026-10-18 · depends on scikit-learn, numpy

1 Recall · 5 min

2 Concept · 10 min