Neural networks, deep learning, NLP · insight-1433
Embed numerical features before the MLP when a tabular network trails gradient boosting
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A plain MLP on raw standardised numerical features of a tabular problem (fare, days to departure, load factor) underperforms a gradient-boosted model by a clear margin.
Action
Add a per-feature numerical embedding (piecewise-linear encoding over quantile bins, or periodic features), retune width and learning rate, and compare with the baseline on the same folds.
Reason
Gorishniy and colleagues report that embedding numerical features, with piecewise linear or periodic schemes, substantially improves tabular deep models and lets simple MLPs approach attention-based ones.
Boundary
Mixed numerical and categorical tabular data of medium size where a neural model is required.
Symptom or risk
A raw scalar fed to a linear layer gives the network a smooth starting point per feature, so sharp, threshold-like or periodic effects can be fitted poorly or slowly.
Exceptions
If a tuned boosting model already satisfies latency and accuracy, keep it; see the baseline-first insights.
Trade-offs
More parameters and preprocessing state to version; bins must come from training data only.
Pitfalls
Fitting quantile bins on the full dataset, which leaks validation distribution.
Lessons and concepts
Claims and their sources
Gorishniy et al. report that piecewise linear and periodic embeddings of numerical features substantially improve tabular deep models, and that simple MLP-type models can then match attention-based ones.