Feature engineering and preprocessing · insight-1115
Replace two near-duplicate prices with one level and one gap before interpreting a linear model
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- aviation, retail
Summary: situation, action, reason, boundary
Situation
Own fare and competitor fare move together, both sit in a linear or logistic model, and their coefficients flip sign between retrains.
Action
Reparameterize as a level (own fare or the pair's mean) and a gap (own minus competitor, or log ratio); confirm predictions are unchanged and check coefficient stability across bootstrap refits.
Reason
A linear reparameterization spans the same space, so ordinary least squares predicts the same, but the coefficients now answer separate questions with much less shared variance.
Boundary
Coefficient-reading work on prices, budgets and capacities that co-move.
Symptom or risk
Opposite-signed, high-variance coefficients make the price effect unreadable even though predictions are fine.
Exceptions
Penalized models, whose penalty is not invariant to reparameterization, and tree models, which ignore the algebra.
Trade-offs
Level and gap need clear definitions in the report, and a gap in absolute units can be confusing across markets.
Pitfalls
Interpreting the gap as causal when the competitor reacts to the same demand signal.