Classical ML, validation, metrics, interpretation, debugging · insight-0170

Remember scikit-learn's logistic regression is penalized by default before comparing it with statistics output

Situation
Coefficients from scikit-learn's LogisticRegression differ from statsmodels or R on the same data, or are used for inference.
Action
For inference, fit an unpenalized model (penalty=None or statsmodels) with standard errors; for prediction, tune C by cross-validation on standardized features.
Reason
Penalization trades bias for variance to improve prediction; coefficients are deliberately shrunk and their default strength is arbitrary relative to your feature scales.
Boundary
Any comparison between ML and statistical fits of GLMs; regulatory or scientific reporting.

Symptom or risk

Shrunken coefficients are reported as effect estimates with no standard errors; analysts doubt one of the libraries.

With large samples and few features, the default penalty may change coefficients only slightly; still state it.

Unpenalized fits are interpretable but can be unstable with many or collinear features.

Forgetting that the default penalty interacts with unscaled features, shrinking small-unit features most.

scikit-learn applies regularization to logistic regression by default, which is common in machine learning but not in statistics.
Linear Models (scikit-learn User Guide 1.1)