Skip to content
ML Academy

project-019 · Forecasting · retail

Coherent store, department and item forecasts with hierarchical reconciliation

Store replenishment needs item-store daily forecasts, category managers need department totals, and finance needs state totals. Today each team forecasts its own level and the numbers do not add up. Does reconciliation produce coherent forecasts without losing accuracy at the level each team actually uses?

Licence
unknown - check before redistribution (governed by the Kaggle competition rules, which were not readable without login)
Reference run
none

At a glance

  • Data

    M5 Forecasting - Accuracy (Kaggle) with a no-login synthetic fallback (public) · As reported in the M5 literature: 3,049 products, 10 stores in 3 US states, daily sales from 2011 to 2016; 42,840 series across all aggregation levels (per the M5 results paper). Confirm counts after download.

    Details on data

  • Baseline

    Bottom-up of seasonal naive (same weekday last week) at item-store level, and independent per-level seasonal naive (incoherent) to show what each team does today.

    Details on baseline

  • Validation

    Rolling origin with the last four 28-day windows of history as test folds; base models and covariance estimates use only data before each origin.

    Details on validation

  • Metric

    RMSSE per level (scaled by in-sample naive error), coherence error, and runtime per fold.

    Details on metric

Success criterion

Reconciled forecasts are exactly coherent (checked numerically at every origin) and, on a 28-day-horizon rolling backtest, are no worse than the unreconciled base forecasts on RMSSE at any hierarchy level by more than 2%, while improving at least two of the three levels the stakeholders named (item-store, department-store, state).

Context

The M5 accuracy competition used Walmart unit sales organised in a hierarchy (state, store, category, department, item); its evaluation weighted errors across levels. Reconciliation methods range from bottom-up and top-down to MinT, which combines forecasts from all levels using an estimate of the forecast-error covariance.

Data

Dataset
M5 Forecasting - Accuracy (Kaggle) with a no-login synthetic fallback (public)
Licence
unknown - check before redistribution (governed by the Kaggle competition rules, which were not readable without login)
Size
As reported in the M5 literature: 3,049 products, 10 stores in 3 US states, daily sales from 2011 to 2016; 42,840 series across all aggregation levels (per the M5 results paper). Confirm counts after download.
Verification
identity verified

Access

Kaggle login and acceptance of the competition rules are required to download; the data cannot be fetched anonymously in CI. Do not commit or redistribute the raw files. Fallback without login: the synthetic airline hierarchy (network, market, route, flight: 1 + 3 + 11 + 19 series) from capstone.airline_sim.simulate(), clearly labelled synthetic; the method and checks carry over, the retail conclusions do not.

Grain

Bottom level: item x store x day unit sales. Aggregation levels used here: total, state, store, store x category, store x department, item x store. For CPU budgets, run one state's stores and one category first.

ColumnMeaning
item_id / dept_id / cat_idProduct and its department and category (names to be confirmed in the downloaded files)
store_id / state_idStore and its US state
d_1 ... d_NDaily unit sales in wide format, one column per day
calendar (date, events, SNAP flags)Day index to date mapping and calendar events, known in advance
sell_prices (store, item, week, price)Weekly selling price; only past and current-week prices are available at a forecast origin

EDA and data-quality rules

  • Confirm the aggregation matrix: summing bottom series reproduces every upper-level series exactly on history.

  • Mark leading zeros before an item's first sale per store as not-yet-listed, not zero demand.

  • Detect store-wide zero days (closures) and exclude them from scaling denominators.

  • Check price coverage: weeks with sales but no price row, and price rows with no sales.

Baseline

Bottom-up of seasonal naive (same weekday last week) at item-store level, and independent per-level seasonal naive (incoherent) to show what each team does today.

Features

Base forecasts per level: AutoETS for aggregate levels, and for item-store either AutoETS or a global LightGBM with lags of at least 28 days, calendar events and current price relative to the item's trailing average (price for future weeks is not known unless the pricing plan is shared; state the assumption you use).

Candidate methods

  • MinT with shrinkage covariance on AutoETS base forecasts at all levels (HierarchicalForecast MinTrace mint_shrink)

    Uses information from every level and is the standard optimal-combination approach; shrinkage is needed because the bottom level has far more series than history allows for a full covariance.

  • Bottom-up of a global LightGBM item-store model

    Coherent by construction and uses cross-item information; tests whether a strong bottom-level model makes reconciliation unnecessary.

Validation design

Rolling origin with the last four 28-day windows of history as test folds; base models and covariance estimates use only data before each origin. Report RMSSE per level and per fold; coherence error (max absolute difference between reconciled parent and sum of children) must be zero to numerical precision.

Leakage checks

In-sample residuals for MinT come from fits on pre-origin data only. Price features use the price known at the origin week; future-week prices are excluded unless a documented price plan exists.

Bias and representativeness

Bias per level and per store; MinT can push small item forecasts negative, so report how many reconciled bottom values were negative and how you handled them (clipping breaks exact coherence; say so).

Robustness checks

Re-run with OLS reconciliation and with a different base model at the top levels; check that the ranking of methods is stable across folds.

Technical metrics

RMSSE per level (scaled by in-sample naive error), coherence error, and runtime per fold.

Business interpretation

For the store level, units over- and under-forecast per day; for finance, absolute percentage error of the 28-day state total. Stakeholders read only their own level, so report per level, not one pooled number.

Error slices

  • Hierarchy level (total, state, store, department-store, item-store).

  • Item-store series by intermittency class.

  • Days with calendar events vs ordinary days.

  • Stores with closures or structural shifts in history.

Implementation steps

  • Obtain data under the competition rules (or build the synthetic fallback); write the aggregation matrix and its coherence test.

  • Subset to one state and one category for the first full loop; record runtime.

  • Fit base forecasts per level per fold; store forecasts and in-sample residuals.

  • Reconcile with BottomUp, MinT-shrink and OLS; verify coherence numerically.

  • Score RMSSE per level and slice; tabulate where reconciliation helps and hurts.

  • Recommend a method per stakeholder need and document negative-value handling.

Reproduction

Learner pipeline with the Kaggle download as a manual, documented step (credentials never in code); the synthetic fallback runs end to end without login and is the CI smoke path.

Limitations

  • Licence for the M5 files is not confirmed here; keep raw data and derived item-level files out of public repositories.

  • Unit sales are not demand: stock-outs are invisible, and zero days may be availability problems.

  • MinT's gains depend on covariance estimates that are noisy for thousands of short series.

Production and monitoring

Nightly or weekly batch: base forecasts, reconciliation, coherence assertion, then publish per level with the same forecast version id. Monitor per-level bias and the reconciliation adjustment size (a sudden jump means a base model broke). Fallback: bottom-up of seasonal naive, which is always coherent.

Interview questions

  • Bottom-up, top-down or MinT: which would you choose for 30,000 item-store series and why?

  • Reconciliation made item forecasts negative. What do you do, and what does it cost you?

  • Finance and store operations disagree about which level should be accurate. How do you resolve it with evidence?

Postmortem and alternatives

Likely findings: reconciliation helps aggregate levels and barely changes intermittent item-store series; MinT runtime and memory dominate. Alternative: middle-out from store-department with proportions from recent history, or reconcile only the levels stakeholders consume.

Rubric3 criteria

CriterionMeetsExceeds
CoherenceAggregation matrix tested; reconciled forecasts coherent to numerical precision.Documents every place coherence is deliberately broken (clipping) and its size.
Per-level evaluationRMSSE per level and fold for baseline and both candidates.Maps each stakeholder decision to its level and recommends accordingly.
Data access honestyKaggle login requirement and unknown licence stated; fallback labelled synthetic.Repository passes a check that no raw M5 file is tracked.

References4