Feature engineering and preprocessing · insight-1112
Choose Yeo-Johnson over Box-Cox when a feature can be zero or negative
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- finance, aviation
Summary: situation, action, reason, boundary
Situation
A PowerTransformer with method box-cox was fitted on fare or revenue columns, and later extracts contain zero fares, refunds or negative adjustments.
Action
Use yeo-johnson for features with zeros or negatives; if Box-Cox stays, enforce a pre-transform check that every value is strictly positive and route failures to the data-quality alert.
Reason
Box-Cox is defined only for strictly positive data, while Yeo-Johnson has separate branches for non-negative and negative values.
Boundary
Linear and distance-based models on skewed money, count and duration features.
Symptom or risk
Transform raises errors or NaNs at serving, or someone adds an ad hoc offset that quietly changes the fitted lambda.
Exceptions
Features that are positive by contract and checked at ingestion.
Trade-offs
Yeo-Johnson is less interpretable than a plain log, and lambda is estimated from training data only.
Pitfalls
Standardizing before the power transform on a feature that must stay positive.
Lessons and concepts
Claims and their sources
Box-Cox can only be applied to strictly positive data; Yeo-Johnson is defined piecewise for non-negative and negative values (scikit-learn section 8.3.2).