Neural networks, deep learning, NLP · insight-0434
Beat a tuned gradient-boosting baseline before shipping a neural net on tabular data
- Evidence type
- empirical finding
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A team proposes a deep model for a tabular problem, such as predicting final bookings per flight from booking-curve, fare and calendar features.
Action
Train LightGBM or XGBoost on the same split and metric first, with a modest tuning budget. Require the neural model to beat it by a margin larger than seed-to-seed variation.
Reason
On medium-sized tabular benchmarks, tree-based models have repeatedly outperformed deep models while needing less tuning.
Boundary
Tabular data of thousands to hundreds of thousands of rows with heterogeneous features.
Symptom or risk
Without a strong tree baseline, the neural model is judged against a weak reference; the project takes on GPU cost and tuning effort for no measurable gain.
Exceptions
Inputs with text, images or long sequences, or very large datasets, where neural models can win; an ensemble of both can also help.
Trade-offs
Running two model families costs time, but the baseline is usually faster to build.
Pitfalls
Tuning the neural net for weeks and the boosting model for minutes.
Lessons and concepts
Claims and their sources
Grinsztajn et al. found tree-based models remain state of the art on medium-sized tabular data (about 10K samples) across 45 datasets.
Shwartz-Ziv and Armon found XGBoost outperformed recent deep tabular models, needed much less tuning, and that an ensemble of both did best.