SQL, data modeling, data quality, EDA, visualization · insight-1072
Set data-test severity by blast radius and keep the failing rows
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data analyst, Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A pipeline has forty dbt-style tests that all fail the build, and a harmless null in an optional column blocks the daily revenue refresh.
Action
Classify tests: error for grain, referential and not-null on keys and money; warn with a threshold for soft expectations. Store failing rows for inspection, name an owner per test and review warnings weekly. Treat a test that never fails as a candidate for removal.
Reason
A test is a query returning violating rows, so it can be tuned by consequence and can carry its evidence.
Boundary
dbt, Great Expectations-style suites and SQL assertions in CI.
Symptom or risk
If every test is blocking, teams learn to disable tests; if none are, a broken key flows into reports, and either way failures arrive without the offending rows.
Exceptions
Regulated data where every constraint is blocking by policy.
Trade-offs
Warning thresholds need maintenance and can mask slow drift.
Pitfalls
Setting thresholds so loose that a warning never fires.
Lessons and concepts
Claims and their sources
In dbt a data test is a select returning failing rows, generic tests include unique, not_null, accepted_values and relationships, severity can be warn or error with warn_if and error_if thresholds, and store_failures saves failing rows to a table.