Time series and forecasting · insight-1214
Put a sampling error on measured interval coverage before calling a model miscalibrated
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Data analyst
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
An 80% interval covered 74% of 50 backtest origins and the team wants to retune the model.
Action
Compute the binomial standard error of the hit rate, sqrt(p(1-p)/n), and judge the gap against it. Collect more origins, pool across series carefully, or accept a wide verdict.
Reason
Coverage is an average of 0/1 hits, so its precision depends on the number of independent forecasts, which is small for long horizons and few origins.
Boundary
Any claim about interval or quantile calibration from a finite backtest.
Symptom or risk
The gap is within sampling noise, so retuning chases noise, while a real miscalibration on a longer record can hide behind a small sample.
Exceptions
With thousands of independent series-origins the error is small and tiny gaps become detectable.
Trade-offs
Waiting for more origins delays decisions; use a rolling record with a stated minimum count.
Pitfalls
Counting overlapping multi-step forecasts as independent hits (see the insight on dependent hits).
Example
Hypothetical: with n = 50 and p = 0.8 the standard error is 5.7 points, so 74% is about one standard error low. With n = 500 it is 1.8 points.