Gradient boosting

For squared error the negative gradient is the residual, so each tree fits what the ensemble still gets wrong. A learning rate scales each tree's contribution; smaller rates need more rounds and usually generalize better. Rounds, learning rate and tree depth jointly set capacity, and the number of rounds is normally chosen by early stopping on validation data. XGBoost, LightGBM and CatBoost implement this idea with different split-finding and categorical-handling choices.