Neural networks, deep learning, NLP · insight-1721
Realign word-level labels to subword tokens before token classification and score only the labelled positions
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A tagger for flight numbers or airport names is trained on word-level labels, but the tokenizer splits rare or inflected words into pieces and adds special tokens.
Action
Map tokens to words with the tokenizer's word index, give special tokens the loss's ignore value, label each word's first piece, ignore the rest, and score the labelled positions only.
Reason
The Hugging Face guide documents this: subword splitting and special tokens break the word-to-label match, and ignored positions are dropped before scoring.
Boundary
Named-entity recognition, part-of-speech tagging and slot filling with subword tokenizers.
Symptom or risk
Labels shift against tokens after the first split word, so the model learns shifted targets and scores compare sequences of different lengths.
Exceptions
Character-level or word-level models without subword splitting; tasks that label every piece by design.
Trade-offs
Predictions come from the first piece only, which can be less accurate for long split words.
Pitfalls
Truncating after labelling so a word's label is lost; reading span-level scores from token positions.
Example
Hypothetical: 8 words become 12 tokens plus 2 special tokens. Only 8 positions carry labels and 6 are ignored; without realignment every label after the first split word sits on the wrong token.
Lessons and concepts
Claims and their sources
The Hugging Face token classification guide realigns word labels to subword tokens by mapping tokens to words, giving special tokens the ignore label -100, labelling only the first token of each word, and dropping -100 positions before computing seqeval metrics.