Classical ML, validation, metrics, interpretation, debugging · insight-1171draft — in review
Treat an exploding coefficient on a rare category as separation and decide the remedy before reaching for a penalty
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist
- Industries
- banking, healthcare, aviation
Summary: situation, action, reason, boundary
Situation
An unpenalized logistic regression gives a coefficient or standard error in the tens of log-odds for a dummy or a bin, or raises a perfect separation error.
Action
Cross-tabulate the outcome against the offending feature, then choose: merge sparse levels, drop or bin the feature, or fit with a ridge penalty. Refit and check that predictions on the non-separated rows barely move.
Reason
The likelihood keeps rising as the coefficient grows, so only merging or shrinkage gives a finite answer.
Boundary
Explanatory or audit-facing models fitted with statsmodels or any unpenalized solver.
Symptom or risk
A level with no events (or a threshold that splits all cases) has no finite maximum-likelihood estimate; the optimizer stops at an arbitrary large value, so its odds ratio and p-value mean nothing.
Exceptions
Pure prediction with a penalized model may be fine; the damage is to interpretation.
Trade-offs
Merging levels loses detail; a penalty shrinks every coefficient, not only the broken one.
Pitfalls
Raising max_iter, which only moves the stopping point of a divergent fit.
Example
Hypothetical: a route-by-fare-class dummy with 0 no-shows in 35 flights gets a coefficient of about -18 and a standard error larger than that.
Lessons and concepts
Claims and their sources
statsmodels defines a PerfectSeparationError for perfect prediction in discrete models, and a separate PerfectSeparationWarning.
scikit-learn's logistic regression applies regularization by default; no regularization corresponds to a very large C.