Feature engineering and preprocessing · insight-1121
Use character n-grams for short, noisy strings such as place names, codes and product titles
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, Applied AI
- Industries
- e-commerce, aviation
Summary: situation, action, reason, boundary
Situation
You classify or match short strings with typos and spelling variants, such as 'Antalya' against 'Antlya' or truncated product names.
Action
Build TF-IDF on character n-grams inside word boundaries (char_wb, range 2 to 4), train a linear model or nearest-neighbour search, and compare with word tokens on a validation set with realistic typos.
Reason
Strings that differ by one character still share most of their character n-grams, so similarity degrades gracefully instead of dropping to zero.
Boundary
Entity matching, address and city normalisation, product-title classification, noisy ticket subjects.
Symptom or risk
Word tokens never match across spelling variants, so near-identical strings look unrelated and most vocabulary entries are singletons.
Exceptions
Long clean documents, where word features are cheaper and more interpretable.
Trade-offs
Many more features and weaker interpretability; char_wb is less noisy than char for whitespace-separated languages.
Pitfalls
Fitting on training strings only and forgetting that serving strings may carry different normalisation such as accents or case.
Lessons and concepts
Claims and their sources
Character n-grams are described as resilient against misspellings and derivations, and char_wb generates significantly less noisy features than char for languages that separate words with white space.