Statistics, experimentation, causal reasoning · insight-0021
Do not filter or control on a variable caused by both the exposure and the outcome
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, Data analyst
- Industries
- aviation, banking, healthcare
Summary: situation, action, reason, boundary
Situation
An analyst studies only customers who filed a complaint, or only flights that sold out, and finds a surprising association between two drivers.
Action
Draw a small DAG before choosing filters and covariates. If a variable is caused by both the exposure and the outcome, or by their causes, do not select, stratify or adjust on it; analyse the unselected population or model the selection explicitly.
Reason
Conditioning on a collider opens a non-causal path between its causes, which biases the estimate even when no confounding exists.
Boundary
Complaint data, approved-loan data, converted-user analyses and any dataset filtered by an outcome-related event.
Symptom or risk
Selecting on a common effect (a collider) creates an association that does not exist in the full population, such as on-time performance looking unrelated or negatively related to satisfaction among complainers.
Exceptions
When the population of interest is defined by the selection and the question is descriptive, the filter is legitimate if stated.
Trade-offs
Using unselected data may require costly joins or new collection.
Pitfalls
Adding every available covariate "to be safe", which can include colliders.
Lessons and concepts
Claims and their sources
Conditioning on a collider (by controlling, stratifying or selecting on it) produces endogenous selection bias, which DAGs make visible.