r11 results - cost sharing and doctor visits

Recorded run of cd python && uv run python -m reference.r11_cost_sharing_two_part.run (full mode). Every number below is copied from the files in outputs/ written by that run (metrics.json, baseline_rates.csv, rate_ratios.csv, agreement_adjusted_vs_unadjusted.csv, interval_comparison.csv, cv_summary.csv, count_distribution_check.csv, model_fit_slices.csv, pricing_table.csv, hurdle_parts.csv, hurdle_decomposition.csv, slice_effects.csv, robustness_by_site.csv, robustness_by_year.csv, balance.csv). The data are the RAND Health Insurance Experiment subset bundled with statsmodels; the dollar cost per visit used in pricing_table.csv (100 USD) is hypothetical and only illustrates the translation to cost.

Item	Value
Run date	2026-10-12
Seeds	CV folds 20261012 (+ repeat index); bootstrap spawned from 20261013 (200 person-level resamples)
Python / numpy / pandas / scipy / statsmodels	3.12.13 / 2.5.3 / 3.0.6 / 1.18.1 / 0.15.0
Runtime of the recorded run	91.9 s inside run.py (95.0 s wall); 10 joblib workers, one BLAS thread each; Apple M5, 10 cores, 16 GB
Smoke mode	--smoke: 1 CV repeat, 8 bootstrap draws, 28.6 s inside run.py (31.9 s wall)
Data	20,190 person-years, 5,912 persons, 6 sites, 18 plans; years 1 to 5 have 5,638 / 5,575 / 5,548 / 1,715 / 1,714 rows; no missing values in the used columns; source file SHA-256 fe64f3c8...5c541
Rows per person	3 rows for 3,743 persons, 5 for 1,584, 1 for 269, 2 for 246, 4 for 70; every person stays in one plan cell
Convergence	all 15 CV fits per model converged; full-data fits converged; bootstrap failed draws: 0 / 0 / 0 (Poisson / NB / hurdle)
Alignment checks	modelling frame equals the source file on mdvis, idp, lpi, fmde, physlm, disea, health dummies, and lncoins = ln(coins + 1); the source column logc does not equal ln(coins + 1) for the individual-deductible rows (see Finding 1)

Findings

1. What the data look like. 31.2% of person-years have zero visits, mean visits are 2.86, the variance is 7.09 times the mean and the maximum is 77. Exposure is nearly complete (mean 0.999 of a year); 45 person-years are below one year and 21 below half a year (smallest 0.077), so the offset matters for a handful of rows, not for the headline. disea is documented as a count but 91.1% of its values are non-integer (0 to 58.6, mean 11.2); it is used as a continuous severity index. The six populated coins x idp cells are free care (6,822 person-years), 25% (4,065), 50% (1,401), 95% (2,653), individual deductible with coins = 0 (4,175) and individual deductible with coins = 100 (1,074, site 1 only). In the source file logc equals ln(96) for the individual-deductible plan with coins = 0 and 0 for the plans with coins = 100, which is not what the coins column implies; I did not resolve this against the RAND documentation (the bundled randdesc.txt only labels the columns), so those two cells are named by what the files contain and kept out of the four pricing tiers.

2. The raw means are not monotone, and the reason is composition. Mean visits fall from 3.56 (free) to 2.79 (25%), 2.56 (50%) and 2.11 (95%) in the non-deductible cells, but the coins = 100 deductible cell sits at 2.68, above 95%. That cell exists only at site 1, and 50% plans are offered only at sites 1, 3, 4, 5 and 6, so cell means mix sites. With site in the model the ratio for the coins = 100 deductible cell drops from 0.755 (unadjusted) to 0.571 and the 50% ratio moves from 0.720 to 0.653 (agreement_adjusted_vs_unadjusted.csv). Adding the precision covariates and year then changes no 25/50/95 ratio by more than 0.025 (for example 50%: 0.653 to 0.676). Site is therefore the covariate that matters; the rest only tighten intervals.

3. Headline rate ratios (visit rate versus free care, adjusted, person-clustered 95% intervals).

   plan cell	Poisson	NB2	raw ratio with naive interval
   25% coinsurance	0.769 [0.705, 0.840]	0.733 [0.674, 0.798]	0.784 [0.767, 0.802]
   50% coinsurance	0.676 [0.594, 0.768]	0.660 [0.585, 0.746]	0.720 [0.695, 0.746]
   95% coinsurance	0.637 [0.567, 0.716]	0.602 [0.536, 0.676]	0.594 [0.577, 0.611]
   individual deductible, coins = 0	0.710 [0.649, 0.776]	0.678 [0.623, 0.739]	0.681 [0.666, 0.697]
   individual deductible, coins = 100 (site 1)	0.587 [0.496, 0.694]	0.585 [0.492, 0.696]	0.755 [0.726, 0.785]

   The NB2 ratios are 2 to 6 percent lower than the Poisson ones. Poisson with a log link is consistent for the rate ratio whenever the mean model is right, whatever the variance; NB2 weights observations differently, so a gap between them is a hint that the mean model is not exactly right (the income-tertile misfit in Finding 6 points the same way), not a sampling-error statement. I did not investigate it further.

4. The naive intervals are far too narrow: 3.6 to 4.2 times in log-width for the unadjusted ratios (baseline_rates.csv). For the three tiers the standard error of the log rate ratio is 0.011 / 0.018 / 0.015 under naive Poisson, 0.028 / 0.045 / 0.037 under the Pearson-scaled quasi-Poisson, 0.029 / 0.043 / 0.039 under HC1 robust errors that treat rows as independent, and 0.045 / 0.066 / 0.059 when clustered by person (interval_comparison.csv). Overdispersion explains roughly two thirds of the gap in log terms (Pearson chi-square / df = 5.93 for the Poisson fit, NB2 alpha = 1.21 with interval 1.15 to 1.27), repeated years per person the rest. Clustering on the inferred family key (2,528 families; the bundled subset has no family id, so this is a heuristic built from consecutive person ids, plan and family size) widens the standard errors further to 0.052 / 0.077 / 0.070, and clustering on the 6 sites (a stress test with too few clusters to trust) gives 0.069 / 0.086 / 0.050. If the randomization unit was the family, as the project record suggests it may have been, the person-clustered intervals shown above are somewhat too narrow; the family-clustered version of the 95% tier is [0.556, 0.730] against [0.567, 0.716].

5. Only the count model that separates "any visit" reproduces the zero share. On person-grouped held-out folds (5 folds x 3 repeats) the predicted zero share is 0.101 for adjusted Poisson, 0.319 for adjusted NB2 and 0.313 for the adjusted hurdle model, against 0.312 observed. The held-out log-likelihood per person-year is -3.023 (Poisson), -2.131 (NB2) and -2.107 (hurdle). Paired differences with a person-level standard error: NB2 over Poisson +0.893 per person-year (z = 20.0), hurdle over NB2 +0.024 (z = 11.1); adding the covariates to NB2 over arm-only NB2 +0.051 (z = 12.3). The spread across the three repeats is below 0.001. Under the rule fixed before the run (within 2 pp pooled, then best log-likelihood) the chosen model is the hurdle. Success criterion 2 (pooled zero share within 2 percentage points) is met: gap +0.008 pp for the hurdle, +0.63 pp for NB2.

6. The hurdle's near-zero gap is partly mechanical; the slices are the real test. The logit part contains one dummy per plan cell, so it reproduces each cell's zero share almost by construction (worst plan cell, c50: -0.07 pp). NB2 misses 95% coinsurance by -6.5 pp (arm-only NB2 by -6.7 pp). The informative checks are slices the model did not condition on as separate parameters. By income tertile the hurdle's held-out zero share is off by -3.1 pp (low), -1.2 pp (middle) and +4.4 pp (high), and observed mean visits are 3.29 against 3.06 predicted in the high tertile and 2.53 against 2.69 in the low one. So the criterion holds pooled and by plan cell, and does not hold within the two outer income tertiles. Log income enters only linearly in both parts; I did not try a more flexible income term. By child/adult, baseline health, and study year the gaps are within 0.05 pp (fair or poor health: +0.04 pp for the hurdle, -2.6 pp for NB2).

7. Where the effect sits: mostly in whether anyone visits. Hurdle parts (odds ratio for any visit; rate ratio for visits given any): 25%: 0.603 [0.533, 0.683] and 0.786 [0.709, 0.873]; 50%: 0.487 [0.406, 0.586] and 0.735 [0.633, 0.853]; 95%: 0.330 [0.287, 0.378] and 0.798 [0.693, 0.918]. Standardized to the whole sample, the probability of at least one visit is 0.784 under free care, 0.696 (25%), 0.654 (50%) and 0.570 (95%), a ratio of 0.73 at 95%; mean visits given any visit go from 4.49 to 3.78, 3.60 and 3.81 (ratio 0.85 at 95%). The conditional ratio is flat from 50% to 95% while the any-visit ratio keeps falling.

8. Pricing table (chosen hurdle model, expected visits per 1,000 member-years, exposure = 1 year, whole-sample covariate mix, person-level cluster bootstrap with 200 draws, 95% percentile intervals).

   plan cell	visits per 1,000 member-years	95% interval	ratio to free care	95% interval
   free care	3,613	[3,449, 3,782]	1.000
   25% coinsurance	2,723	[2,546, 2,928]	0.754	[0.701, 0.823]
   50% coinsurance	2,450	[2,150, 2,718]	0.678	[0.592, 0.772]
   95% coinsurance	2,288	[2,060, 2,573]	0.633	[0.566, 0.720]
   individual deductible, coins = 0	2,550	[2,350, 2,735]	0.706	[0.657, 0.770]
   individual deductible, coins = 100 (site 1)	2,107	[1,816, 2,455]	0.583	[0.494, 0.694]

   With the hypothetical 100 USD per visit, expected outpatient visit cost per member-year falls from 361 USD to 272, 245 and 229 USD for the three tiers (differences of -89, -116 and -133 USD). The intervals of the 25%, 50% and 95% tiers overlap; I did not compute paired differences between tiers, so the ordering among them is suggestive, while the difference of each tier from free care is well established.

9. Model choice moves the pricing table by a few percent, not by much more. Free-care visits per 1,000 member-years are 3,583 (Poisson), 3,709 (NB2) and 3,613 (hurdle); the 25% tier 2,756, 2,720 and 2,723; the 95% tier 2,282, 2,232 and 2,288. The ratio to free care for the 95% tier is 0.637, 0.602 and 0.633. Differences between models are smaller than the bootstrap interval half-widths (about 250 visits at 95%).

10. Heterogeneity. Joint Wald tests that the three coinsurance ratios do not vary by slice give p = 0.37 (child versus adult), 0.080 (baseline health), 0.091 (income tertile) and 0.72 (study year 1 versus 2 to 5). Nothing is significant at 5%, and the two borderline ones are exactly where the data are thin: 1,862 person-years have fair or poor baseline health and the 95% ratio there is 0.942 [0.685, 1.294] against 0.599 [0.530, 0.677] for good or excellent health. The 25% ratio is 0.793, 0.674 and 0.824 in the low, middle and high income tertiles: no gradient in the direction "cost sharing binds more for low-income members". With four slice families and several ratios each, I read the differences as noise rather than a pattern.

11. Robustness. By site (arm-site cells that exist) the 25% ratio ranges from 0.585 (site 6) to 0.907 (site 2), the 95% ratio from 0.524 (site 6) to 0.715 (site 5); site 4's 50% ratio is 0.978 [0.729, 1.312] and site 5's is 0.409 [0.215, 0.777], i.e. single sites are noisy and their intervals are wide. By study year the 25% ratio runs from 0.696 to 0.811, no trend. Dropping the 45 partial-year rows changes the ratios by at most 0.011 (50%: 0.665 against 0.676). Restricting to the four sites that offer all four tiers (3 to 6, 11,692 person-years) gives 0.694 / 0.681 / 0.612, close to the pooled 0.769 / 0.676 / 0.637 for 50% and 95% but lower for 25%.

12. Balance is acceptable within site except for family size. The largest within-site standardized mean difference against free care is -0.21 (family size, individual-deductible cell with coins = 0); the other values above 0.1 are family size in the 50% cell (-0.18) and in the coins = 100 cell (-0.12) and "poor" self-rated health in the coins = 100 cell (-0.10). Pooled over sites the chronic-disease index differs by up to 0.32 SD between cells, which shrinks to 0.08 within site: this is site composition, not a randomization failure. I cannot tell from these files whether the family-size differences reflect randomization at the family level or subsetting.

13. Leakage demonstration. Adding same-year log outpatient expenditure (outpdol, an outcome of the plan) to the Poisson model moves the rate ratios from 0.769 / 0.676 / 0.637 to 0.948 / 0.908 / 0.967, with the 25% and 95% intervals now covering 1. It is shown only to size the damage; the headline models do not include it.

Success criteria of project-003

#	Criterion	Result
1	Rate ratios for each coinsurance level versus free care with 95% intervals that allow for overdispersion and person clustering	Met: Poisson-robust and NB2 ratios with person-clustered intervals for all five non-free cells (Finding 3)
2	Chosen count model reproduces the observed zero share within 2 percentage points on person-grouped held-out folds	Met pooled (+0.008 pp) and in every plan cell (worst -0.07 pp). Not met within the income tertiles (+4.4 pp high, -3.1 pp low), which the criterion did not ask for but which is reported (Finding 6)
3	Pricing table with intervals per tier and a statement of how far the experimental population and era limit transfer	Table: Finding 8. Transfer limits: next section

What this does not prove

• Transfer to today's members. The experiment is historical (the bundled documentation gives no dates; the RAND literature places it in the 1970s and early 1980s) and covers six sites; prices, care delivery (telehealth, urgent care, specialist access) and benefit design have changed. The pricing table says what visit rates were under those plans then, not what a current member will do under a 25%, 50% or 95% plan. The effect of a deductible plan, copayments, or out-of-pocket maxima is not identified by coinsurance level alone.
• The randomization unit is not verified. The bundled subset has no family id, so I clustered by person and, as a check, by an inferred family key and by site. If plans were assigned to families, the person-clustered intervals are slightly too narrow (Finding 4).
• Visits only. Spending, admissions, health outcomes and welfare effects are not modelled. A visit-rate reduction says nothing about whether the visits avoided were low or high value.
• Subset and attrition. The bundled subset is not the published analysis file; participation and attrition by plan cannot be checked (rows per person are 1 to 5, with 1,715 and 1,714 rows in years 4 and 5), and I assume the subset is representative of the full experiment.
• The model class is limited. All three candidates use a log-linear mean with linear income, age and health terms. The income-tertile misfit (Finding 6) shows that this is not exact; the hurdle's logit also uses log(time) as a covariate rather than an offset, which is a modelling choice, not an identity.
• Intervals are conditional on the model form. The bootstrap resamples persons and refits the same model class; it does not include model-selection uncertainty (the chosen model won on the full sample, not inside each draw). Percentile endpoints from 200 draws are noisy at the second digit.
• Pricing standardizes to this sample. "Per 1,000 member-years" uses the covariate mix of the experiment, one full year of eligibility and no interaction between plan and covariates; a different membership mix gives different totals even if the ratios hold.
• Slice and robustness results are not tuned and not corrected for multiplicity. Single-site and single-year estimates have wide intervals and are shown to show instability, not to rank sites.
• The 100 USD cost per visit is invented for illustration.

Reproduce

cd python && uv run python -m reference.r11_cost_sharing_two_part.run          # full run, about 90 seconds
cd python && uv run python -m reference.r11_cost_sharing_two_part.run --smoke  # about 30 seconds, writes to a temp dir
cd python && uv run pytest tests/test_r11_cost_sharing_two_part.py             # 12 tests