Skip to content
ML Academy

project-005 · Analytics and statistics · healthcare

Readmissions by admitting specialty: case-mix-adjusted comparison without chasing noise

A hospital network's quality team wants to review discharge planning in the admitting specialties whose 30-day readmission rates are above what their patient mix predicts. Which specialties are credibly above expectation, and which only look high because they are small or treat sicker patients?

Licence
CC BY 4.0 (as stated on the UCI page)
Reference run
none

At a glance

  • Data

    Diabetes 130-US Hospitals for Years 1999-2008 (UCI 296) (public) · 101,766 encounters x 50 columns; diabetic_data.csv 18.3 MB plus IDS_mapping.csv 2.5 KB

    Details on data

  • Baseline

    Crude 30-day readmission rate per specialty ranked in a league table, with Wilson intervals.

    Details on baseline

  • Validation

    Case-mix model: patient-grouped 5-fold CV; out-of-fold predictions are used for expected counts so the model does not grade its own training data.

    Details on validation

  • Metric

    Case-mix AUC, Brier, calibration slope/intercept; O/E with intervals; BH-adjusted q-values; flag overlap between halves.

    Details on metric

Success criterion

  1. Observed/expected (O/E) readmission ratio per specialty with 95% intervals, using an expected-risk model that excludes specialty.
  2. A funnel plot of O/E against expected count with 95% and 99.8% limits, and a flagged list with false discovery rate controlled at 10% (Benjamini-Hochberg).
  3. The case-mix model's out-of-fold calibration slope lies between 0.9 and 1.1.
  4. Stability: at least 70% of flagged specialties are flagged again when the analysis is repeated on two random halves of patients (split by patient, not encounter); otherwise the memo says the flags are unstable.

Context

The UCI data cover ten years (1999-2008) of diabetic inpatient encounters from 130 US hospitals and integrated delivery networks, with inclusion criteria set by the providers (inpatient, diabetic, length of stay 1-14 days, labs and medications given). There is no hospital identifier, so the comparison unit is the admitting specialty. Patients appear in several encounters, and some discharge dispositions make readmission impossible (for example death); the code meanings are in IDS_mapping.csv. Small specialties produce extreme crude rates by chance, which is why league tables mislead.

Data

Dataset
Diabetes 130-US Hospitals for Years 1999-2008 (UCI 296) (public)
Licence
CC BY 4.0 (as stated on the UCI page)
Size
101,766 encounters x 50 columns; diabetic_data.csv 18.3 MB plus IDS_mapping.csv 2.5 KB
Verification
identity verified

Access

Direct download (zip about 3 MB) from the UCI page or ucimlrepo.fetch_ucirepo(id=296); no account. Attribute definitions are in Table 1 of Strack et al. (2014), which the UCI page cites. Fallback: none needed.

Grain

One row per inpatient encounter (encounter_id); patient_nbr repeats across encounters. Primary analysis: first eligible encounter per patient, which removes within-patient dependence; a sensitivity analysis keeps all encounters with patient-clustered errors.

ColumnMeaning
encounter_id / patient_nbrEncounter key and patient key
age10-year age band
admission_type_id / admission_source_idCoded admission type and source (see IDS_mapping.csv)
discharge_disposition_idCoded discharge destination; used to exclude deaths and hospice
medical_specialtySpecialty of the admitting physician; the comparison unit; often missing
time_in_hospitalDays between admission and discharge (1-14)
number_inpatient / number_emergency / number_outpatientVisits in the year before the encounter
diag_1Primary diagnosis, first three ICD-9 characters (hundreds of values; group by chapter)
A1CresultHbA1c result band or not measured
readmitted<30, >30 or NO; outcome = readmitted within 30 days

EDA and data-quality rules

  • Missing-value tokens: the UCI metadata lists the missing symbol as NaN, but scan categorical columns for placeholder strings (for example '?' or 'Unknown/Invalid' in gender) and treat them explicitly.

  • Exclude encounters whose discharge disposition precludes readmission (death, hospice), using code meanings from IDS_mapping.csv; report counts.

  • Patient repetition: count encounters per patient and fix the first-encounter rule before computing any rate.

  • Specialty missingness: report the share with missing specialty; keep it as its own group in descriptions and exclude it from flagging.

  • Group primary diagnoses into ICD-9 chapters with a documented mapping; assert every code maps.

  • Minimum volume: specialties below a pre-set expected count are shown but never flagged.

Baseline

Crude 30-day readmission rate per specialty ranked in a league table, with Wilson intervals. It shows the problem the project solves: the extremes are mostly small specialties.

Features

Expected-risk (case-mix) model inputs, all known at discharge: age band, sex, admission type and source, discharge disposition group, time in hospital, prior-year inpatient/emergency/outpatient visits, number of diagnoses, primary diagnosis chapter, HbA1c measured, insulin use and medication change. Specialty is excluded (otherwise the model adjusts away the differences being compared). Race is handled in a sensitivity analysis.

Candidate methods

  • Logistic regression expected-risk model with indirect standardization (O/E per specialty)

    Interpretable and standard for risk adjustment; calibration is easy to check.

  • Gradient-boosted expected-risk model

    Tests whether a more flexible case-mix model changes which specialties are flagged; if not, the simpler model is defended.

  • Random-intercept logistic model for specialty

    Shrinks small specialties toward the mean and gives specialty effects with uncertainty in one model.

Validation design

  • Case-mix model: patient-grouped 5-fold CV; out-of-fold predictions are used for expected counts so the model does not grade its own training data. Report AUC, Brier score, calibration slope and intercept.
  • Flag stability: repeat the full pipeline on two random patient halves and report overlap of flagged sets.
  • Multiple comparisons: Benjamini-Hochberg across all eligible specialties at 10% FDR.

Leakage checks

Use only information available at discharge. The >30 label is part of the outcome variable and is never a feature. Out-of-fold expected risks prevent overfitted expectations.

Bias and representativeness

Adjusting for race can mask disparities in care; report flags with and without it. Coding practices differ across hospitals and years, and the data cannot show readmissions captured elsewhere (unknown); flags are prompts for review, not verdicts.

Robustness checks

First-encounter versus all encounters (clustered); logistic versus boosted expected model; volume threshold variations; 95% versus 99.8% funnel limits.

Technical metrics

Case-mix AUC, Brier, calibration slope/intercept; O/E with intervals; BH-adjusted q-values; flag overlap between halves.

Business interpretation

Number of specialties flagged for review, and an upper-bound count of excess readmissions if flagged specialties moved to O/E = 1 (associational, labelled as an upper bound).

Error slices

  • Age bands: is calibration worse for the oldest patients?

  • Emergency admissions versus referrals.

  • HbA1c measured versus not measured.

  • Primary diagnosis chapters with high readmission (for example circulatory).

Implementation steps

  • Download and checksum; load with placeholder tokens mapped to missing; join code meanings from IDS_mapping.csv.

  • Apply exclusions and the first-encounter rule; produce a cohort flow table.

  • Build the crude league table baseline.

  • Fit and cross-validate the expected-risk models; check calibration.

  • Compute O/E, funnel limits and BH flags; run the split-half stability check and sensitivity analyses.

  • Write the quality-team memo with the funnel plot and a short guide to reading it.

Reproduction

Learner-built. One command from raw CSV to memo; pinned environment; patient-level split seeds in config; cohort flow counts saved so reviewers can reconcile every exclusion.

Limitations

  • No hospital identifier: specialty comparisons pool very different hospitals.

  • Historical (1999-2008) US data; coding and care pathways have changed.

  • Risk adjustment cannot remove unmeasured severity; flags are associational.

Production and monitoring

Quarterly refresh with the same frozen case-mix model (re-fitted yearly), funnel plots per specialty, and a review workflow where flagged units examine cases before any conclusion. Monitor calibration of the case-mix model on new data; recalibrate when the intercept drifts.

Interview questions

  • Why is ranking specialties by crude readmission rate misleading, and what replaces it?

  • Should race be in a risk-adjustment model for provider comparison? Argue both sides.

  • A specialty is flagged in one patient half and not the other. What do you report?

Postmortem and alternatives

If the goal were reducing readmissions directly, a patient-level risk model for targeting transitional care would be the project, with different metrics (calibration at the top of the risk distribution, capacity). A hierarchical model is the natural alternative to O/E with funnels when there are many small units.

Rubric4 criteria

CriterionMeetsExceeds
Cohort definitionExclusions and first-encounter rule documented in a cohort flow table.Shows how results change with all encounters and clustered errors.
Risk adjustmentCalibrated out-of-fold expected-risk model that excludes specialty.Compares two expected models and a hierarchical model and explains differences in flags.
Multiplicity and noiseFunnel plot and FDR-controlled flags with a volume threshold.Split-half stability analysis and a clear statement of what a flag does and does not mean.
Fairness reasoningReports flags with and without race adjustment.Explains the trade-off and recommends a reporting convention.

References6