Classical ML, validation, metrics, interpretation, debugging · insight-1710
Model an ordered target with ordered thresholds and score it by how many levels it misses
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- cross-industry, aviation
Summary: situation, action, reason, boundary
Situation
The target has ordered levels, such as complaint severity 1 to 5, and you are about to fit an unordered multiclass model and report accuracy.
Action
Fit an ordered logit, or a binary decomposition asking whether the level is above k, and report mean absolute level error and the share off by two or more levels next to accuracy. Compare the above-k logits before trusting one slope vector.
Reason
An ordered logit reads the label as a discretised latent score, so K levels need K-1 thresholds and one slope vector, not K-1 slope vectors.
Boundary
Severity grades, satisfaction scores and risk bands with three or more levels.
Symptom or risk
An unordered model learns separate coefficients per level, wasting data on small samples, and counts a miss by one level like a miss by four.
Exceptions
When effects differ by threshold, one slope vector is wrong; use the binary decomposition.
Trade-offs
Rigid, but few parameters.
Pitfalls
Regressing on level codes as if gaps were equal.
Example
Simulated (latent logistic score from 8 features, 5 levels, 150 training rows, 60 draws): multinomial logistic regression has mean absolute level error 1.01 and accuracy 0.35; ordered logit 0.93 and 0.38, better on error in 59 of 60 draws.
Lessons and concepts
Claims and their sources
statsmodels' OrderedModel treats the endogenous variable as ordered with no numeric meaning beyond the ordering, models it through a latent linear variable observed via cut points, and needs K-1 thresholds besides the slope coefficients for K levels.
Frank and Hall published a simple approach to ordinal classification at ECML 2001; only the bibliographic record was checked, so it is cited for the existence of the approach, not for a result.