Neural networks, deep learning, NLP · insight-0452
Fit the vectorizer vocabulary and IDF weights on training text only
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, Data analyst
- Industries
- e-commerce
Summary: situation, action, reason, boundary
Situation
A notebook calls
TfidfVectorizer().fit_transform(all_texts)and then splits the matrix into train and test.Action
Put the vectorizer inside a pipeline, fit it on the training split (or within each CV fold), and transform validation, test and serving text with the fitted object.
Reason
Vocabulary and IDF are learned statistics, so they follow the same holdout rule as scaling or target encoding.
Boundary
TF-IDF, count vectorizers, BM25 statistics and any vocabulary built from your own corpus.
Symptom or risk
Test-only words enter the vocabulary and test documents shape the IDF weights, so offline scores are optimistic, especially when new products or routes appear only in later data.
Exceptions
Pretrained tokenizers and vocabularies fixed before your data was collected.
Trade-offs
Out-of-vocabulary words in new data are ignored; character n-grams soften that.
Pitfalls
Building one vocabulary across time periods and evaluating on the latest period.
Lessons and concepts
Claims and their sources
scikit-learn's common-pitfalls guide warns that fitting preprocessing on the full dataset before splitting leaks test information and recommends pipelines.
TfidfVectorizer learns vocabulary and IDF from the documents passed to fit.