Time series and forecasting · insight-0251

Count every strategy variant you tried before trusting the best backtest

Situation
A trading or pricing signal looks strong after trying many lookback windows, thresholds and universes on the same history.
Action
Log every configuration tested, hold back a final untouched period (or paper-trade forward), adjust expectations for the number of trials, and prefer parameter regions that perform well broadly over a single sharp peak.
Reason
Selecting the maximum over many noisy evaluations inflates the winner's score; this is the same selection bias that affects model selection and multiple testing.
Boundary
Quant strategies, pricing rules tuned on history, alert thresholds, forecast model searches.

Symptom or risk

The best of many backtests is selected partly for luck, so its reported performance is biased upward and often vanishes live.

A small number of pre-registered variants tested once carries little selection bias.

Holding back data shortens the development sample; forward testing delays deployment.

Reusing the 'untouched' period after a disappointing first look.

Overfitting the model-selection criterion produces optimistically biased performance estimates for the selected model.
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation