Neural networks, deep learning, NLP · insight-0404
Check how many parameters your embedding tables hold before adding ID features
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- retail
Summary: situation, action, reason, boundary
Situation
You add learned embeddings for high-cardinality identifiers such as customer, store or SKU IDs to a neural model.
Action
Compute cardinality times embedding dimension per table, and look at the distribution of examples per ID. Map IDs below a minimum count to a shared unknown bucket and size dimensions to data, not to habit.
Reason
An embedding row receives gradient only when its ID appears in a batch, so rare IDs keep near-random vectors that the model can still use to memorize training rows.
Boundary
Recommendation, retail demand and fraud models with entity IDs.
Symptom or risk
Embedding tables quietly dominate the parameter count, most rows see a handful of examples, and validation loss rises after the first epoch while training loss keeps falling.
Exceptions
When every ID has many examples, or when IDs are the main signal and you validate on the same ID population.
Trade-offs
Bucketing rare IDs loses some personalization; larger tables need more memory and stronger regularization.
Pitfalls
Validating on random rows shares IDs between train and validation and hides the problem; use a group or time split.
Example
Synthetic retail case: 200,000 SKUs x 64 dimensions = 12.8 million parameters against 1.5 million training rows.