r12 results - district math results, a binomial model that separates context from performance

Recorded run of cd python && uv run python -m reference.r12_district_binomial_model.run (full mode). Every number below is copied from the files in outputs/ written by that run (metrics.json, dispersion.csv, pearson_by_size.csv, cv_deviance.csv, ame_table.csv, ame_sequence.csv, slices.csv, robustness.csv, vif.csv). The data are the 1998 California STAR district subset bundled with statsmodels; they are not copied into this repository.

Item	Value
Run date	2026-10-12
Seeds	CV RepeatedKFold 20261012 (10 folds x 5 repeats); bootstrap spawned from 20261013 (200 district resamples); out-of-fold flag folds 20261014
Python / numpy / pandas / scipy / statsmodels	3.12.13 / 2.5.3 / 3.0.6 / 1.18.1 / 0.15.0
Runtime of the recorded run	14.5 s inside run.py (17.7 s wall); Apple M5, 10 cores, 16 GB
Smoke mode	--smoke: 1 CV repeat, 10 bootstrap draws, 9.7 s inside run.py (12.9 s wall)
Data	303 districts, 267,611 tested students, no missing values; tested students per district: min 33, quartiles 187.5 / 445 / 1,020, max 38,852; the 5 largest districts hold 23.0% of all tested students, the 10 largest 29.1%
Pooled share above the national median	0.405 (students); district mean 0.437, SD 0.184, range 0.076 to 0.928
Convergence	all 50 CV fits of every specification and family converged; bootstrap failed draws: 0
Interaction columns	all eight precomputed columns equal the product of their components (max absolute difference 5e-5, rounding in the stored columns)
VIF	largest 3.9 (PERHISP), then 3.5 (PTRATIO), 3.1 (PERMINTE); no variable above 10, so the VIF-pruning robustness variant dropped nothing

Findings

1. The plain binomial model is rejected by a wide margin. Pearson chi-square / df is 109.9 for the intercept-only model, 30.1 with LOWINC, 15.47 for the headline specification (all base variables except PCTAF; 11 predictors, 291 residual df) and 14.70 with PCTAF. Deviance / df is 15.69 for the headline model. Context removes most of the excess, but a factor of about 15 remains between-district variation that the binomial counts cannot produce.

2. A single constant scale (quasi-binomial) is not adequate either. If the extra variation were a constant multiple of the binomial variance, the mean squared Pearson residual would be the same in small and large districts. It rises from 6.98 in the smallest size quartile (33 to 187 students) through 10.09 and 15.25 to 27.12 in the largest (1,029 to 38,852), a factor of 3.9. That pattern is what a between-district random effect produces, because its variance does not shrink with n. The rule fixed before the run therefore chose the observation-level random intercept; its standard deviation on the logit scale is 0.347 for the headline specification (0.850 intercept-only, 0.445 with LOWINC only, 0.339 with PCTAF).

3. Headline average marginal effects (percentage-point change in the share above the median per +1 unit of the variable).

   variable	binomial (scale 1)	quasi-binomial	random intercept (chosen)	bootstrap, random intercept
   LOWINC (per +1 point of low-income share)	-0.396 [-0.413, -0.379]	-0.396 [-0.463, -0.329]	-0.496 [-0.566, -0.427]	[-0.591, -0.424]
   PTRATIO (per +1 pupil per teacher)	-0.398 [-0.570, -0.227]	-0.398 [-1.072, 0.276]	-0.151 [-0.915, 0.612]	[-0.856, 0.544]

   One SD of LOWINC (20.4 points) is -10.1 pp under the chosen model; one SD of PTRATIO (2.2 pupils) is -0.33 pp. The binomial intervals are 3.9 times too narrow for both variables (the square root of 15.47). The binomial model "finds" a significant pupil-teacher effect (interval entirely below zero); the chosen model does not. The student-weighted version of the chosen model gives -0.479 [-0.546, -0.413] for LOWINC and -0.146 [-0.884, 0.592] for PTRATIO. Delta-method intervals are close to the bootstrap ones for LOWINC; for PTRATIO the bootstrap interval is narrower by 0.06 to 0.07 pp at each end.

4. The pupil-teacher estimate is mostly context. Adding variables step by step to a model with PTRATIO, the chosen-model average marginal effect of PTRATIO goes from -1.42 [-2.35, -0.49] (PTRATIO alone) to -0.83 [-1.35, -0.31] (plus LOWINC), -0.35 [-0.83, 0.12] (plus the three race composition shares), -0.15 [-0.92, 0.61] (plus teacher, spending and school-type variables) and -0.02 [-0.78, 0.73] (plus PCTAF). The point estimate shrinks to about a tenth of its starting size and the interval covers zero from the third step on. Under the quasi-binomial model the sequence is -2.57, -1.50, -0.45, -0.40, -0.11. Per-pupil spending is not distinguishable from zero either: +1.60 pp per 1,000 USD [-0.95, 4.15] without PCTAF, +1.34 [-1.17, 3.85] with it. LOWINC is stable across steps 2 to 5 (about -0.4 to -0.5 pp per point).

5. Cross-validated deviance (10 folds x 5 repeats, districts held out, plug-in deviance at the model's mean probability).

   specification (binomial GLM)	total CV deviance	per district	vs intercept-only	vs LOWINC-only	SD over repeats
   intercept only	36,333	119.9			606
   LOWINC only	9,624	31.8	-73.5%		67
   headline (no PCTAF)	5,438	17.9	-85.0%	-43.5%	165
   headline + PCTAF	5,031	16.6	-86.2%	-47.7%	88
   headline + PCTAF + spending x PTRATIO	5,067	16.7	-86.1%	-47.4%	95
   headline + PCTAF + all eight interactions (standardized products)	9,429	31.1	-74.0%	-2.0%	8,939

   Success criterion 3 holds: the headline model is 43.5% below LOWINC-only and 85.0% below intercept-only. LOWINC alone captures most of the gain (73.5%). The pre-specified spending x PTRATIO interaction does not improve prediction (5,067 against 5,031), and the all-interactions model is unstable: the mean over repeats is 9,429 but the median is 5,335, one repeat has a single district contributing 77.7% of the total, and the SD over repeats is 8,939. This is why the project record asks for a small pre-specified set instead of all eight columns. Even for the stable models a single district can account for 7 to 19% of the total deviance (largest single-district share column in cv_deviance.csv), so a few large districts dominate the sum.

6. Scored on the model's own predictive distribution, the random-intercept model is far better than the binomial. The plug-in deviance of the random-intercept models is slightly worse than the binomial GLM's (5,652 against 5,438 for the headline specification), because their fixed effects are fitted with nearly equal weight per district rather than weight proportional to size. Judged by the held-out log-score of the model's own predictive distribution (cv_predictive_deviance), the binomial's 17.9 per district becomes 3.69 for the random-intercept headline model, with PCTAF 3.66, LOWINC-only 4.03 and intercept-only 5.22. A binomial distribution with a mean this accurate is wildly over-confident about each district's count.

7. Which districts fall outside the limits depends entirely on the variance model. Out of fold, 185 of 303 districts (61.1%) lie outside the plain binomial 95% limits; 20 (6.6%) outside quasi-binomial limits and 18 (5.9%) outside random-intercept limits, against about 15 expected if the model were right. In sample the counts are 179, 14 and 13. All 18 random-intercept flags are also flagged by the binomial limits. The 18 flagged row positions are in metrics.json (flagged_row_positions_random_intercept_oof); district identities are not recorded because the data are not redistributed. By size quartile the random-intercept flag rate is 13.2% in the smallest, 2.6%, 5.3% and 2.6% in the largest; the smallest quartile has 76 districts, so 10 flags there is above the roughly 4 expected and I did not explain it (the model's limits may be too tight for small districts, or the mixing distribution is not normal). By LOWINC quartile the flag rates are 3.9%, 9.2%, 4.0% and 6.6%.

8. Residual spread follows the random-intercept model, not the binomial. The standard deviation of the out-of-fold residual (observed minus predicted share, in pp) is 11.8, 8.0, 7.9 and 6.4 from the smallest to the largest size quartile. A binomial model would give 4.9, 2.8, 1.9 and 1.2; the random-intercept model implies 8.9, 7.9, 7.9 and 7.5. The mean residual in every size and LOWINC quartile is within +-2.1 pp. Districts with any charter schools (35 of them) have a mean residual of +2.9 pp (observed above predicted); districts with at least 25% year-round students (52) +0.2 pp.

9. Robustness (quasi-binomial logit, robustness.csv). Probit and cloglog links change the average marginal effects by at most 0.03 pp (LOWINC -0.399 and -0.393 against -0.396; PTRATIO -0.380 and -0.370 against -0.398). One district with 38,852 tested students has Cook's distance 15.5, the next (8,479 students) 8.7; 131 districts exceed 4/n. Dropping the five most influential districts moves LOWINC to -0.463 and PTRATIO from -0.398 to +0.006 [-0.761, 0.772], so the sign of the pupil-teacher estimate is not stable. Equal weight per district gives -0.498 and -0.230. Removing the race composition shares moves PTRATIO to -1.139 [-1.981, -0.297] and LOWINC to -0.638: the pupil-teacher "effect" is recovered exactly when part of the context is left out.

Success criteria of project-007

#	Criterion	Result
1	Binomial GLM with a defended treatment of overdispersion	Met: Pearson chi-square / df = 15.47; constant scale rejected by the size-quartile check (3.9x); random intercept chosen (Findings 1 and 2)
2	Average marginal effects for low-income share and pupil-teacher ratio with 95% intervals under the chosen variance model	Met (Finding 3). The pupil-teacher interval covers zero
3	10-fold CV binomial deviance versus intercept-only and low-income-only	Met (Finding 5)
4	Written statement of what the model cannot support	Met: next section

What this does not prove

• Nothing individual. Every estimate is a district-level association (303 districts, 1998, California). A negative LOWINC coefficient does not mean a low-income student scores lower because of income, and district-level slopes can differ in size and sign from student-level ones (the ecological fallacy). No student-level relationship is estimated here.
• Nothing causal about policy levers. Pupil-teacher ratio and spending are tangled with context; the estimate for PTRATIO moves from -1.42 to -0.15 pp as context is added and flips sign when five districts are dropped. Even the final interval [-0.92, 0.61] pp per pupil is wide enough to include effects a department would care about in both directions. "Not significant" is not "no effect".
• No rankings. A district's out-of-fold flag is a statement that its share is unusual given 12 variables and a normal random intercept; a district below its limit may be mediocre, and one above it may be affected by a variable that is not in the data (for example English-learner share). Intervals on the district effects themselves were not computed.
• PCTAF is ambiguous. Course-taking may be a consequence of performance (reverse causation); I report the specification with and without it, but I cannot say which is "right". Choosing the headline without PCTAF is a pre-specified convention, not evidence.
• The random-intercept model is one parametric choice. A logit-normal mixing distribution was assumed, not tested; the excess flags in the smallest districts (Finding 7) hint that it is imperfect. The marginal effects average over this distribution.
• Marginal effects are averages over these districts' covariate values with equal weight per district (a student-weighted version is in ame_table.csv); they do not apply to a particular district.
• Cross-validation deviance is dominated by a few large districts, and the CV is at district level, not student level; it does not tell how the model would predict next year.
• Exploratory choices. The PCTAF-free headline is a convention; the equal-weight, no-race-shares and Cook's-distance variants probe the pupil-teacher estimate and are exploratory, not pre-specified. The nested sequence in Finding 4 is an ordering I chose; a different order would give different intermediate numbers.
• A subset of districts and variables. The bundled file is described as a subset of the original data (the documentation calls the cases both districts and counties; the long description says unified school districts, and the code and this document say "district"). The original data are subject to the author's reserved rights, so they are not redistributed.

Reproduce

cd python && uv run python -m reference.r12_district_binomial_model.run          # full run, about 15 seconds
cd python && uv run python -m reference.r12_district_binomial_model.run --smoke  # about 10 seconds, writes to a temp dir
cd python && uv run pytest tests/test_r12_district_binomial_model.py             # 9 tests