Classical ML, validation, metrics, interpretation, debugging · insight-1157
Lower the learning rate and let early stopping choose the tree count
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A boosted model with learning rate 0.3 and 200 trees is tuned by hand on several other parameters first.
Action
Fix a small learning rate such as 0.05 to 0.1, set a generous tree limit, let early stopping on a validation set choose the count, then tune depth and regularization.
Reason
Smaller learning rates need more trees to reach the same training error, and empirical evidence suggests they give better test error.
Boundary
Gradient boosting: scikit-learn, LightGBM, XGBoost, CatBoost.
Symptom or risk
A large step size reaches a worse optimum and interacts with the tree count, so every other parameter is tuned against a moving target.
Exceptions
A higher rate is fine for quick exploration or feature screening, where speed matters more than the last bit of accuracy.
Trade-offs
Training time and model size grow roughly in inverse proportion to the rate.
Pitfalls
Tuning depth and regularization at one learning rate, then lowering the rate and keeping the parameters without a recheck.
Lessons and concepts
Claims and their sources
In scikit-learn gradient boosting, learning_rate strongly interacts with n_estimators, smaller values need more weak learners to keep the same training error, empirical evidence suggests small values favor better test error, and the docs recommend learning_rate <= 0.1 with enough estimators that early stopping applies.