Classical ML, validation, metrics, interpretation, debugging · insight-1161
Treat huge logistic coefficients and unstable standard errors as a sign of separation
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- healthcare, banking
Summary: situation, action, reason, boundary
Situation
A logistic regression returns coefficients of 15 or 30 for a dummy variable, enormous standard errors or a convergence warning.
Action
Cross-tabulate the flagged feature with the outcome, collapse sparse categories, then use a penalty or Firth's bias-reduced estimator; do not just raise max_iter.
Reason
With separated data a finite maximum likelihood estimate does not exist; penalization restores finite estimates.
Boundary
Small samples, rare outcomes and sparse categorical features in statsmodels or unpenalized scikit-learn fits.
Symptom or risk
One or more features perfectly separate the classes in the sample, so the likelihood keeps rising as the coefficient grows; reported effects and p-values are meaningless.
Exceptions
For pure prediction, a penalized model is usually fine; the problem concerns coefficients and inference.
Trade-offs
Penalized estimates are biased by design; Firth's method keeps inference possible with profile intervals.
Pitfalls
Dropping the offending category quietly, which removes the strongest real signal from the model.
Lessons and concepts
Claims and their sources
The maximum likelihood estimate in logistic regression does not exist when the data are separated (Albert and Anderson, 1984).
Firth's bias-reducing modification of maximum likelihood is a likelihood penalized by the Jeffreys prior; its use under data separation, where the maximum likelihood estimate does not exist, is described in the later literature (cited here at abstract level, Firth's paper itself not opened).