Classical ML, validation, metrics, interpretation, debugging · insight-1711
Fit a logit-link model to a bounded share such as load factor instead of a straight line
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- aviation, retail
Summary: situation, action, reason, boundary
Situation
The target is a share between 0 and 1, such as load factor, and a linear regression predicts values below 0 or above 1.
Action
Fit a GLM with a logit link on the proportion itself (statsmodels' Binomial family accepts proportions), use robust standard errors, and compare with the clipped linear model.
Reason
The logit link keeps predictions inside (0, 1) and flattens them towards the bounds. The Binomial family fixes the scale at 1, so default standard errors assume binomial variance that shares rarely have.
Boundary
Load factors and mix shares with many values strictly inside (0, 1).
Symptom or risk
Impossible predictions get clipped by hand, and a line fits worst near the bounds.
Exceptions
Many values exactly at 0 or 1 may need a two-part model.
Trade-offs
Effects on the share depend on the starting level.
Pitfalls
Taking the logit of the observed share, which fails at exactly 0 and 1, or averaging shares without weighting by the denominator.
Example
Simulated (beta-distributed shares, mean logistic in 3 features, 400 rows, 40 repeats): linear regression predicted outside [0, 1] for 9% of test rows, with mean absolute error 0.124 (0.113 clipped); the logit GLM had 0.102.
Lessons and concepts
Claims and their sources
The statsmodels Binomial family accepts a 1d array of proportions as endog, with trial counts passed as var_weights, and its default link is the logit.
In statsmodels GLM the scale is fixed at 1 for the Binomial family.
Papke and Wooldridge address econometric methods for fractional response variables; only the bibliographic record was read.