SQL, data modeling, data quality, EDA, visualization · insight-0057

Check whether missingness depends on other columns before dropping incomplete rows

Situation
An analysis drops rows with missing fare or revenue because "only 3% are missing".
Action
Tabulate the missing rate by segment, time and source system. If it varies with observed variables, use weighting or imputation conditioned on them, add missingness indicators, and report results with and without the missing rows.
Reason
Simple deletion is generally safe only when data are missing completely at random; under MAR or MNAR it biases estimates.
Boundary
Revenue, survey, sensor and CRM fields with partial coverage.

Symptom or risk

If missingness is concentrated in one segment, complete-case results underweight that segment and shift the mean and the trend.

Very low missingness that is uniform across segments, verified rather than assumed.

Imputation models add assumptions and must be fitted on training data only.

Testing the missingness pattern only on the overall table, where segment differences cancel.

Synthetic airline data: avg_fare and revenue are missing at random conditional on market in flight_day_raw, more often on intl-long flights.

MCAR, MAR and MNAR are distinct missing-data mechanisms, and simple fixes such as deletion are generally valid only under MCAR.
Flexible Imputation of Missing Data (2nd edition, online)