SQL, data modeling, data quality, EDA, visualization · insight-0060
Encode impossible values as explicit integrity rules that quarantine rows and count them
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data analyst, ML engineer, Data scientist
- Industries
- aviation, manufacturing, energy
Summary: situation, action, reason, boundary
Situation
EDA finds load factors above 1 and negative bookings in a raw table that feeds a forecast.
Action
Write rules for each domain limit (0 <= load_factor <= 1, bookings >= 0, passengers <= capacity), run them on every load, move failing rows to a quarantine table with the rule name, and alert when counts exceed a baseline.
Reason
Rules turn analyst knowledge into repeatable checks; counting failures over time separates a single glitch from a broken source.
Boundary
Any table with physical, accounting or business limits.
Symptom or risk
One-off fixes in a notebook leave the pipeline unprotected, so the next bad batch flows into dashboards and model training.
Exceptions
Values that look impossible but are legitimate, such as overbooked flights with load factor above 1 on a passengers-booked basis; define the measure precisely first.
Trade-offs
Strict rules can block loads for minor issues; use warn and error tiers.
Pitfalls
Clipping impossible values to the limit silently, which hides the source problem.
Example
Synthetic airline data: flight_day_raw has 6 rows with load_factor above 1 and 2 rows with negative bookings.