Feature engineering and preprocessing · insight-1120

Lowercase Turkish text with language-aware rules, because default lowercasing splits dotted and dotless I

Situation
Turkish free text, such as complaint notes or city names, is lowercased by the default step of a TF-IDF vectorizer.
Action
Normalise the text before vectorizing with explicit rules: map capital dotted I to i and capital I to dotless i for Turkish fields, strip stray combining marks, then lowercase; test on a list of known city and carrier names.
Reason
Python lowercases by the Unicode default conversion, which is not language-specific: 'I' becomes 'i' rather than the Turkish dotless i, and 'İ' becomes 'i' plus a combining dot, two code points.
Boundary
Turkish and Azeri text; also a general check for any language with special casing rules.

Symptom or risk

The same word appears as several tokens, vocabulary and idf statistics fragment, and rare-word features lose support.

Text that is already normalised upstream, or tasks using a multilingual subword tokenizer that handles case itself.

Language detection per field is needed on mixed-language data; wrong rules damage other languages.

Applying Turkish rules to English product names in the same column.

Observed in Python 3: 'I'.lower() returns 'i', and the length of 'İ'.lower() is 2.

Python's str.lower follows the lowercasing algorithm of section 3.13.2 (Default Case Conversion) of the Unicode Standard.
Built-in Types: str.lower and str.casefold (Python 3 documentation)