Neural networks, deep learning, NLP · insight-1441
Print what a stock stop-word list removes before applying it to complaint or sentiment text
- Evidence type
- empirical finding
- Difficulty
- easy
- Roles
- Data scientist, Data analyst
- Industries
- aviation, retail
Summary: situation, action, reason, boundary
Situation
A pipeline strips the packaged English stop-word list before TF-IDF on airline complaints because the tutorial did.
Action
Print the words the list deletes from a sample of your texts, compare with and without it (or use
max_df) under cross-validation, and keep negations and domain words.Reason
Nothman et al. compared 52 English lists and found omissions, entries that may not belong, and clashes with tokenizers; scikit-learn's own docs warn of known issues with its "english" list.
Boundary
Bag-of-words models on short opinionated or complaint text.
Symptom or risk
Lists contain words that carry label signal, such as negations; two texts with opposite meaning can collapse to the same features and the model cannot separate them.
Exceptions
Long documents with a topical label may lose nothing; test rather than assume.
Trade-offs
Keeping all words enlarges the vocabulary and the noise in a linear model.
Pitfalls
Using one language's list for mixed-language text; filtering before building n-grams so that "not" disappears from bigrams.
Example
Checked locally with scikit-learn 1.9.1: with stop_words="english", "bag was not delivered" and "bag was delivered" both become the features "bag" and "delivered".
Lessons and concepts
Claims and their sources
Nothman, Qin and Yurchak compare 52 popular English stop lists and report omissions, entries that may not belong, and clashes with tokenizers.
scikit-learn's TfidfVectorizer docs warn of several known issues with the built-in "english" stop-word list.