Neural networks, deep learning, NLP · insight-0456
Inspect how the tokenizer splits your domain codes and numbers
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, ML engineer, Data scientist
- Industries
- aviation, healthcare
Summary: situation, action, reason, boundary
Situation
Inputs contain flight numbers, fare-class codes, booking references, part numbers or ICD codes, and a pretrained model handles them poorly.
Action
Print tokenizations for a sample of codes. Normalize formats, add separators, or map codes to descriptive text or structured features; consider domain-adaptive pretraining if code-heavy text dominates.
Reason
Subword tokenizers are learned from general text, and tokenizer quality affects downstream performance, not only sequence length.
Boundary
Domain-heavy text in aviation, healthcare, manufacturing and finance.
Symptom or risk
Codes are split into arbitrary subword pieces, so related codes share no tokens and the model cannot reuse what it learned; errors cluster on code-heavy inputs.
Exceptions
Codes that never carry meaning for the label can be masked or dropped.
Trade-offs
Adding tokens or retraining a tokenizer requires training new embeddings and can break checkpoints.
Pitfalls
Concluding the model lacks domain knowledge when the issue is input formatting.
Lessons and concepts
Claims and their sources
Sennrich et al. introduced byte-pair-encoding subword units that segment rare words into frequent pieces.
Rust et al. find a dedicated monolingual tokenizer plays a role in downstream performance comparable to pretraining data size.