project-009 · Classical ML · healthcare

Predicting one-year diabetes progression with an honest error estimate: nested CV on 442 patients

A clinic wants to schedule follow-up intensity from baseline measurements. Can ten baseline variables predict the one-year disease-progression score well enough to sort patients into follow-up tiers, and what error should the clinic expect on new patients?

Dataset: Diabetes dataset (sklearn.datasets.load_diabetes)
Licence: unknown - check before redistribution (not stated for the data; scikit-learn code is BSD-3-Clause)
Reference run: executed
Lessons: day 11 day 12 day 15 day 17 day 18

Data
Diabetes dataset (sklearn.datasets.load_diabetes) (bundled-with-library) · 442 rows x 10 features + target (observed when loading, scikit-learn 1.9.1)
Details on data
Baseline
Two baselines under the same nested CV: DummyRegressor predicting the training-fold mean, and ordinary least squares on all ten features.
Details on baseline
Validation
Nested CV: outer RepeatedKFold(nsplits=5, nrepeats=5) for the estimate, inner KFold(5) inside GridSearchCV for tuning. The model family is chosen by the outer estimate.
Details on validation
Metric
Outer-loop MAE and RMSE (mean and spread), R-squared as a secondary summary, optimism of inner scores, bootstrap sign stability of linear coefficients.
Details on metric

Success criterion

Nested cross-validation (outer 5-fold repeated 5 times, inner 5-fold) estimates MAE and RMSE for every candidate pipeline, with the spread across outer folds reported.
The selected approach beats the mean-prediction baseline by at least 15% relative MAE in the outer estimate (a pre-stated, hypothetical usefulness bar); if it does not, the report recommends against model-based tiers.
No pre-registered residual slice has MAE above 1.25 times the overall MAE without an explanation.
An interpretation section states what coefficients and importances can and cannot support.

Context

scikit-learn bundles the diabetes data from Efron et al. (2004): 442 patients, ten baseline variables (age, sex, BMI, blood pressure and six serum measurements) and a quantitative measure of disease progression one year later. With 442 rows, a single train/test split is noisy and tuning on the same folds used for reporting is optimistic; nested CV is the point of the project. Observed when loading: s1 and s2 (total cholesterol and LDL) correlate at 0.90, so their individual coefficients are unstable.

This dataset ships with scikit-learn and needs no download. California housing, the other common choice, is fetched over the network on first use (fetch_california_housing downloads an archive from figshare and caches it), which is why the reference implementation uses diabetes.

Data

Dataset: Diabetes dataset (sklearn.datasets.load_diabetes) (bundled-with-library)
Licence: unknown - check before redistribution (not stated for the data; scikit-learn code is BSD-3-Clause)
Size: 442 rows x 10 features + target (observed when loading, scikit-learn 1.9.1)
Verification: download verified

Access

load_diabetes(as_frame=True) for mean-centred, scaled features or load_diabetes(as_frame=True, scaled=False) for original units; files ship inside scikit-learn, so no download and no account.

Grain

One row per patient at baseline; target measured one year later.

Column	Meaning
age	Age in years (scaled by default)
sex	Coded 1 or 2 in the unscaled data; the coding is not documented, so do not label the groups
bmi / bp	Body mass index; average blood pressure
s1 / s2 / s3	Total serum cholesterol; LDL; HDL
s4 / s5 / s6	Total cholesterol / HDL; possibly log of serum triglycerides; blood sugar
target	Quantitative measure of disease progression one year after baseline (25-346 when loaded)

EDA and data-quality rules

Decide scaled versus unscaled features explicitly; scaling inside a Pipeline is fitted on training folds even though the bundled default is pre-scaled on all 442 rows (state this minor leakage in the report).
Correlation matrix of the serum variables; flag pairs above 0.8 before interpreting coefficients.
Target distribution and its relation to each feature (scatter with smoother); look for non-linearity in bmi and s5.
The sex coding is undocumented: report results by 'group 1' and 'group 2' only.

Baseline

Two baselines under the same nested CV: DummyRegressor predicting the training-fold mean, and ordinary least squares on all ten features. OLS is a strong baseline at this sample size; any extra complexity has to beat it.

Features

The ten provided variables, standardized inside the pipeline. Optional pre-specified additions: squared bmi and bmi x bp. No feature selection outside the pipeline; if lasso drops features, that happens inside each fold.

Candidate methods

Regularized linear models (ridge, lasso, elastic net) with the penalty tuned in the inner loop
Handles the s1/s2 collinearity and small n; coefficients stay interpretable with caveats.
Gradient boosting (HistGradientBoostingRegressor) with shallow trees and strong regularization
Tests whether non-linearity or interactions add value; with 442 rows its variance may cancel any gain, which is itself a finding.

Validation design

Nested CV: outer RepeatedKFold(n_splits=5, n_repeats=5) for the estimate, inner KFold(5) inside GridSearchCV for tuning. The model family is chosen by the outer estimate. The final model is refit with inner-loop tuning on all data; its expected error is the nested estimate, not its in-sample error. Also report the optimism: inner best score versus outer score.

Leakage checks

Tuning and reporting on the same folds (non-nested) is the main leak; also scaling or feature selection outside the pipeline. The bundled default features were scaled using all rows; using scaled=False with in-fold scaling avoids that.

Bias and representativeness

Single historical cohort with unknown sampling; the undocumented sex coding prevents subgroup claims; the target scale is not clinically defined in the documentation.

Robustness checks

Different outer seeds; MAE versus RMSE ranking; removing the two most influential patients; scaled versus unscaled inputs.

Technical metrics

Outer-loop MAE and RMSE (mean and spread), R-squared as a secondary summary, optimism of inner scores, bootstrap sign stability of linear coefficients.

Business interpretation

Tier agreement: share of patients placed in the same follow-up tier (for example tertiles of predicted progression) as their actual tier, out of fold; expected misplacements per 100 patients.

Error slices

Slices by predicted value (low/medium/high), not by true target: conditioning on the true target always shows regression to the mean.
BMI tertiles.
Age tertiles.
Sex group 1 versus group 2 (unnamed).

Implementation steps

Load unscaled data; run the EDA rules; write the dictionary and the sex-coding caveat.
Build pipelines (scaler + model) for baselines and candidates with their grids.
Run nested CV with fixed seeds; collect outer-fold scores and inner best parameters.
Compare families on the outer estimate; compute optimism; choose and refit.
Residual slices out of fold; bootstrap coefficient stability; permutation importance on held-out folds.
Write the report: honest error, tier agreement, interpretation limits.

Reproduction

Reference (specified, not yet implemented): cd python && uv run python -m reference.r03_tabular_regression.run. Runs offline because the data ship with scikit-learn. Seeds for both CV loops in config; dependencies pinned by python/uv.lock; target runtime under 2 minutes on a laptop CPU.

Limitations

442 patients from one source; error estimates are themselves uncertain, which the outer-fold spread shows.
Associational model: coefficients and importances do not say what changing BMI or cholesterol would do.
The target's clinical meaning and the sex coding are undocumented.

Production and monitoring

Not ready for clinical use from this analysis alone: it would need prospective validation on the clinic's own patients, a model card, documented tier thresholds, monitoring of input distributions and of realised progression by tier, and governance appropriate to clinical decision support.

Interview questions

Why is the best inner-loop CV score an optimistic estimate of the final model's error?
Lasso dropped s2 in most folds. Can you conclude LDL is irrelevant to progression?
Your residuals are large and positive for patients with high true progression. Is the model biased for them?

Postmortem and alternatives

If tiers are the real decision, an ordinal or quantile model of progression could target the tier boundaries directly. California housing would have exercised spatial structure and larger n, at the cost of a network download.

Reference implementation

Status: executed — outputs below are from the recorded run

cd python && uv run python -m reference.r03_tabular_regression.run

r03 results - diabetes progression: an honest error estimate with nested cross-validation

Recorded run of cd python && uv run python -m reference.r03_tabular_regression.run (full mode: outer RepeatedKFold 5 x 5, inner KFold 5). Every number below is copied from outputs/metrics.json, outputs/nested_cv_summary.csv, outputs/residual_slices.csv and outputs/coefficients_and_importance.csv, written by that run. The data are sklearn.datasets.load_diabetes(as_frame=True, scaled=False): 442 patients, ten baseline variables, one-year progression score. The data ship with scikit-learn (no download); their licence is not stated in the documentation (unknown, check before redistributing).

Item	Value
Run date	2026-10-11
Seeds	outer CV 20261011, inner CV 20261012, boosting / bootstrap / permutation 20261013; robustness outer seeds 101, 202, 303
Python / numpy / pandas / scipy / scikit-learn	3.12.13 / 2.5.3 / 3.0.6 / 1.18.1 / 1.9.1
Runtime of the recorded run	81.7 s wall (84.4 s by `time`; 311 s user and 200 s system CPU) on a 10-core laptop shared with other jobs (load average about 13). An earlier run of the same code, with identical results, took 137.8 s at a load average of about 28, so the 2-minute target depends on free cores. The nested CV itself took about 15 s.
Rows / features / target	442 / 10 / progression score 25 to 346 (mean 152.1, sd 77.1)
Outer folds	25 per candidate (5 x 5), about 354 training and 88 test patients each

Findings

Model-based prediction clears the pre-stated bar by a wide margin. The mean-prediction baseline has an outer MAE of 66.0; the selected pipeline (ridge with the two pre-specified terms, bmi^2 and bmi x bp, alpha = 1.0 on the full-data refit) has 43.8, a relative gain of 33.7% against the hypothetical 15% bar. The corrected resampled paired t-test (Nadeau and Bengio) for the difference of 22.2 MAE gives t = 9.1, p = 2.8e-9.

Candidate	Outer MAE, mean (sd over 25 folds)	Outer RMSE, mean (sd)	R-squared	Inner best MAE	Outer minus inner MAE
dummy mean (baseline)	66.01 (3.77)	77.18 (3.81)	-0.020	undefined (no tuning)	undefined
OLS (baseline)	44.23 (3.59)	54.65 (3.59)	0.484	undefined (no tuning)	undefined
ridge	44.23 (3.65)	54.65 (3.58)	0.484	44.66	-0.43
lasso	44.26 (3.66)	54.66 (3.58)	0.484	44.66	-0.40
elastic net	44.23 (3.63)	54.62 (3.56)	0.485	44.66	-0.43
ridge + bmi^2 + bmi x bp	43.76 (3.72)	54.25 (3.54)	0.492	44.24	-0.48
histogram gradient boosting	45.59 (2.87)	56.49 (2.83)	0.449	45.45	+0.14

The linear family is a statistical tie. The selected pipeline beats OLS by 0.47 MAE (1.1%; corrected t = 1.56, p = 0.13), ridge by 0.47 (p = 0.13), lasso by 0.50 (p = 0.12) and elastic net by 0.46 (p = 0.14). Ridge, elastic net, OLS and lasso lie within 0.03 MAE of each other, and their order differs between MAE and RMSE. The fold-to-fold sd is about 3.6 MAE, and the sd of the five per-repeat means is 0.15 to 0.30. What the evidence does support is "linear models beat the mean by about a third" and "no candidate beats OLS by a margin that survives the correction".
Boosting does not help at n = 442. It is worse than the selected pipeline by 1.83 MAE (p = 0.095, weak evidence) and worse than OLS on both MAE and RMSE. Its fold spread is the smallest of the models (sd 2.9), consistent with a smoother but biased fit; this was not tested further.
The selection effect is visible only for the larger grid. Outer MAE minus inner best-score MAE is negative for the four linear pipelines (-0.40 to -0.48), the opposite of the textbook "inner best score is optimistic", and +0.14 for boosting. The likely reason is that inner folds train on about 283 rows against about 354 in the outer loop, which inflates the inner error more than selecting among seven alphas deflates it; this was not isolated by a separate experiment. The non-nested contrast (best CV score when tuning and reporting on all data, three shuffles) agrees: nested minus non-nested is -0.09 to -0.13 for the linear pipelines (no optimism detectable) and +0.61 for boosting (44.98 non-nested against 45.59 nested; optimism of 0.61 MAE on an 8-combination grid).
Tier agreement is modest. Using tertiles of the out-of-fold predictions against tertiles of the actual progression, the selected pipeline places 57.8% of patients in the same tier (42.2 of 100 misplaced, 3.8% two tiers off), against 33.3% expected by chance. OLS has the same mean agreement (57.8%; the per-repeat counts differ but their sum is equal). The mean baseline's 31.2% is arbitrary (its "tiers" come from fold-specific means) and is shown only as a floor.

Residual slices (out-of-fold, overall MAE 43.76). No slice exceeds the 1.25x bar, so criterion 3 is met; the largest ratio is 1.15.

Slice (formed on predictions or pre-treatment features)	MAE	Ratio to overall
Predicted value low / medium / high tertile	31.1 / 50.4 / 49.9	0.71 / 1.15 / 1.14
BMI low / medium / high tertile	36.2 / 47.5 / 48.2	0.83 / 1.09 / 1.10
Age low / medium / high tertile	43.6 / 45.4 / 42.2	1.00 / 1.04 / 0.96
Sex group 1 / group 2 (coding undocumented; n about 235 / 207)	45.2 / 42.1	1.03 / 0.96

Errors are smaller where the predicted progression is low (about 31) and about 50 where it is medium or high.

Regression to the mean is an artefact of grouping by the true target. Mean residual (actual minus prediction) by tertile of the actual target is -39.8 (low), -5.4 (medium) and +45.5 (high); by tertile of the prediction it is +1.5, +0.6 and -2.1. The first pattern appears for any noisy predictor and is not evidence of bias against high-risk patients (see outputs/residuals.png, right panel).
Robustness. The selected pipeline is first by MAE under outer seeds 101, 202 and 303 (two repeats each; second place varies between OLS and elastic net) and after dropping the two most influential patients (rows 123 and 382, Cook's distance 0.021 and 0.024; MAE 43.91). Feeding the bundled pre-scaled variant instead of the raw units changes MAE by at most 1e-14 for the mean baseline, OLS, ridge, lasso and elastic net, by 0.014 for boosting and by 0.114 for the product-term pipeline (43.69 scaled, 43.80 unscaled; one outer repeat), because squares and products of uncentred variables are not affine-invariant.
The final model's in-sample MAE (42.63) is not its expected error. The expected error of the same procedure on new patients is the nested estimate, 43.76 (with a fold sd of 3.7 and a corrected standard error for differences of about 0.3). The gap is only 1.1 MAE here because the model is low-capacity.

Interpretation limits

Coefficients are standardized (change in the progression score per one standard deviation of the variable), from the selected pipeline refitted on all patients; "share" columns come from 300 bootstrap refits at the full-data hyperparameters; permutation importance is the increase in held-out MAE when a feature is shuffled (5 repeats per outer fold, averaged over the 25 folds). Full tables are in outputs/coefficients_and_importance.csv.

Feature	Standardized coef. (product-term ridge)	Bootstrap share positive	OLS coef. (bootstrap median)	OLS share positive	Lasso: share of folds at exactly 0	Permutation importance, product-term ridge / OLS
age	+0.6	0.58	-0.7	0.43	0.24	-0.02 / -0.04
sex	-11.4	0.00	-11.3	0.00	0.00	2.21 / 2.10
bmi	-31.7	0.03	+24.4	1.00	0.00	8.26 / 8.10
bp	-15.5	0.06	+15.3	1.00	0.00	3.78 / 3.60
s1	-30.9	0.03	-36.8	0.03	0.00	10.51 / 16.99
s2	+19.2	0.94	+21.7	0.93	0.28	4.49 / 6.72
s3	+0.9	0.53	+4.8	0.69	0.08	0.34 / 0.59
s4	+6.5	0.83	+8.3	0.87	0.20	0.72 / 1.01
s5	+33.5	1.00	+34.7	1.00	0.00	13.91 / 16.12
s6	+3.3	0.89	+3.2	0.87	0.04	0.14 / 0.06
bmi^2	+21.4	0.82	not in OLS	not in OLS	not in lasso	not computed
bmi x bp	+55.0	1.00	not in OLS	not in OLS	not in lasso	not computed

The bmi and bp signs flip when the product terms are added. In plain OLS both are positive in 100% of bootstrap refits; in the product-term model they are negative in 97% and 94%, because bmi^2 and bmi x bp are built from raw units and are strongly collinear with bmi and bp. The main-effect coefficients of that pipeline are therefore not interpretable on their own; only the fitted surface and the predictions are.
s1 and s2 are a collinear pair (r = 0.897, the only pair above 0.8). They take large coefficients of opposite sign (about -31 to -37 and +19 to +22), both stable in sign across bootstrap refits. Lasso zeroes s2 in 28% of outer folds and never zeroes s1, so "lasso dropped LDL" would be true in only some folds and does not show that LDL is irrelevant.
Permutation importance rises for a collinear pair. Shuffling s1 breaks its relation with s2 and creates combinations that do not occur in the data (s1 importance 17.0 in OLS against 10.5 in the product-term model), so the numbers show how much the fitted function uses a variable, not how much the outcome depends on it.
Everything here is associational. The coefficients do not say what lowering BMI or cholesterol would do to progression, and the bootstrap holds the hyperparameters fixed at the full-data value.

What this does not prove

It does not show that this model is safe or useful for scheduling patients. It is 442 patients from one source; the target scale and the sex coding are undocumented; there is no prospective validation, no tier thresholds defined by a clinician (tertiles of this sample were used) and no governance.
It does not show that the two pre-specified terms matter. Their gain over OLS (0.47 MAE, p = 0.13) is within the noise of the repeated-CV comparison, and the pipeline was selected because it had the lowest outer MAE among five candidates; the selection effect on that reported 43.76 is not quantified here.
It does not show that boosting cannot beat linear models on these data; only that an 8-combination shallow grid did not, at n = 442.
The 25 outer folds overlap across repeats, so their sd is not a standard error of the mean, and the Nadeau-Bengio correction is an approximation.
The "inner score is pessimistic because of less training data" reading in finding 4 is an explanation consistent with the numbers, not a tested one.
Tier agreement of 58% against 33% by chance says the ranking is informative, not that a 3-tier follow-up policy would improve outcomes.
The runtime depends on free cores (81.7 s and 137.8 s on the same machine at different loads).

Reproduce

cd python && uv run python -m reference.r03_tabular_regression.run
cd python && uv run pytest tests/test_r03_tabular_regression.py

Builder-authored reference. It is not learner evidence and is never credited to your portfolio.

Rubric4 criteria

Criterion	Meets	Exceeds
Evaluation design	Nested CV with pipelines and fixed seeds; family chosen on the outer estimate.	Quantifies optimism and shows how the conclusion changes with a non-nested design.
Model comparison	Baselines and two candidate families compared on MAE and RMSE with spread.	Explains why the simpler family wins or loses using bias-variance reasoning at n = 442.
Residual analysis	Slices by predicted value and features, not by true target.	Explains the regression-to-the-mean artefact with a plot.
Interpretation limits	States that coefficients are associational and unstable under collinearity.	Shows bootstrap sign stability and permutation importance with the correlated-feature caveat.

References7

Diabetes dataset (sklearn.datasets.load_diabetes)
Data description and access.
Least angle regression
Original publication that used these data.
Bias in error estimation when using cross-validation for model selection
Why tuned CV error is biased and nested CV is needed.
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation
Overfitting in model selection.
Cross-validation: evaluating estimator performance (scikit-learn User Guide 3.1)
RepeatedKFold and the other CV iterators; nested CV is built by wrapping GridSearchCV inside an outer cross_val_score loop.
Regularization and Variable Selection Via the Elastic Net
Elastic net for correlated predictors.
Permutation feature importance (scikit-learn User Guide 5.2)
Held-out importance and its correlated-feature caveat.

At a glance