Neural networks, deep learning, NLP · insight-1439
Normalise Unicode text to one form before tokenising, hashing or deduplicating
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Turkish messages arrive from several apps and keyboards, and the same letter with a cedilla or breve is sometimes one precomposed character and sometimes a base letter plus a combining mark.
Action
Apply one normalisation form (NFC for most text) at ingestion, before hashing, vectoriser fitting and tokenisation, and count distinct code point sequences before and after.
Reason
Canonically equivalent sequences can have different binary forms; normalising both sides to one form makes comparison reliable.
Boundary
User-generated text from mixed sources, especially copy-paste and mobile input.
Symptom or risk
The strings look identical but compare unequal, so vocabulary entries split, dictionary lookups miss, and deduplication hashes treat copies as different texts.
Exceptions
NFKC also folds compatibility characters such as ligatures and full-width forms and can change meaning; check before using it.
Trade-offs
A small preprocessing cost; a pretrained tokenizer may already normalise, so avoid doing it twice differently.
Pitfalls
Normalising the training set but not the serving path; keeping raw and normalised text without recording which is which.
Example
In Python, "ç" (c plus combining cedilla) is not equal to "ç", but normalising it to NFC makes them equal.
Lessons and concepts
Claims and their sources
UAX 15 defines canonical equivalence, states that equivalent strings can differ in binary form, and defines NFC (decompose then recompose) and NFD (full decomposition).