Neural networks, deep learning, NLP · insight-1442

Use feature hashing when the vocabulary keeps growing, and accept what it gives up

Situation
A ticket classifier is refit nightly and a fitted vocabulary keeps meeting new product names, flight codes and typos, while the vocabulary dictionary grows in memory.
Action
Try a hashing vectoriser with a power-of-two number of columns, compare it with the vocabulary model on the same splits, and record the loss of readable feature names.
Reason
Hashing is stateless: no vocabulary is stored and new tokens get a column. scikit-learn notes the signed hash lets collisions tend to cancel, advises a power of two, and states there is no inverse transform.
Boundary
Streaming or frequently refit text models and very large token spaces.

Symptom or risk

Tokens unseen at fit time are silently dropped at serving, and the vocabulary object must be versioned and shipped with every model.

A stable vocabulary where coefficients must be read as words is better served by a fitted vocabulary.

Collisions add noise; no per-word coefficient lists; IDF must be learned separately.

Too few columns; expecting a feature-importance table of words.

Hypothetical: 2**20 columns give about 1.05 million weights per class, roughly 8 MB in float64, regardless of how many distinct tokens appear.

scikit-learn documents feature hashing with a signed hash so collisions tend to cancel, advises a power-of-two n_features, and notes that inverse_transform is not available.
Feature extraction (scikit-learn User Guide 8.2)