Neural networks, deep learning, NLP · insight-1438
Test character n-grams before building a stemmer for agglutinative Turkish text
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- aviation, telecom
Summary: situation, action, reason, boundary
Situation
A Turkish complaint classifier uses word-level TF-IDF, and one stem shows up in many surface forms because suffixes stack onto it.
Action
Fit a character n-gram TF-IDF (for example
char_wb, 2 to 5 grams) beside the word model and compare both with cross-validation before investing in a stemmer or lemmatiser.Reason
Character n-grams share pieces between inflected forms and misspellings. Bojanowski et al. built word vectors from sums of character n-grams to cover rare and unseen words, noting the value for morphologically rich languages, and scikit-learn documents character n-grams as resilient to misspellings and derivations.
Boundary
Turkish, Finnish, Hungarian and other suffix-heavy languages; short, noisy texts.
Symptom or risk
Every inflected form becomes its own sparse feature, counts thin out, and a form first seen in production has no weight at all.
Exceptions
A well-tested lemmatiser on a large clean corpus may match or beat n-grams; the backtest decides.
Trade-offs
More features and a longer fit; n-gram weights are harder to read than word weights.
Pitfalls
Using char instead of char_wb and learning fragments that straddle word boundaries; skipping the comparison.
Example
Hypothetical: "bagajim", "bagajimi" and "bagajlarimiz" are three separate word features but all contain the 5-gram "bagaj".
Lessons and concepts
Claims and their sources
Bojanowski et al. represent each word as a bag of character n-grams, which handles rare and unseen words and is stated to matter for morphologically rich languages.
scikit-learn documents character n-grams as resilient against misspellings and derivations, with char_wb less noisy than char for whitespace-separated languages.