Classical ML, validation, metrics, interpretation, debugging · insight-1175
Compare Pearson chi-square to residual degrees of freedom before trusting Poisson standard errors
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- insurance, aviation, manufacturing
Summary: situation, action, reason, boundary
Situation
A Poisson regression for counts (no-shows per flight, tickets per shift) returns narrow intervals and very significant drivers.
Action
Compute Pearson chi-square divided by residual degrees of freedom; a value well above 1 signals overdispersion. Refit with an estimated scale (scale='X2' in statsmodels) or a negative binomial model (the GLM family takes alpha as an input; statsmodels' discrete NegativeBinomial estimates it), and compare standard errors and held-out deviance.
Reason
Poisson fixes the dispersion at 1; the negative binomial adds a parameter (alpha) so variance can grow faster than the mean.
Boundary
Count models for event frequencies with unmodelled heterogeneity.
Symptom or risk
Real counts are often more variable than a Poisson with variance equal to its mean allows; the fixed dispersion understates standard errors and overstates significance.
Exceptions
Very few events per cell, where the ratio itself is unreliable.
Trade-offs
A quasi-likelihood scale keeps point estimates but loses a true likelihood; negative binomial changes weights and coefficients modestly.
Pitfalls
Jumping to a zero-inflated model to explain overdispersion with no structural-zero mechanism.
Lessons and concepts
Claims and their sources
statsmodels GLM.fit estimates scale as Pearson chi-square divided by df_resid with scale='X2', and the default scale is 1 for Binomial and Poisson.
The statsmodels NegativeBinomial family has variance mu + alpha mu^2 and fixes the dispersion at 1, while the Poisson variance function leaves no free dispersion.