Engineering, production, governance, cost, communication · insight-1290
Date-stamp the evaluation plan before looking at candidate results
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Five variants and a new feature set are compared with last quarter's champion, with the primary metric, slices and exclusions still open (hypothetical).
Action
Before running, write the hypothesis, primary metric, data window, slices, exclusions, acceptance threshold and variant list; store it with a timestamp; label any later addition as exploratory in the report and list amendments in a change log.
Reason
Fixing the question and analysis plan before outcomes are observed separates testing from hypothesis generation, which hindsight bias otherwise blurs.
Boundary
Model bake-offs, release reviews, vendor evaluations and forecast method comparisons.
Symptom or risk
Metric, slices and exclusions get chosen after the results are visible, so the reported win mixes real signal with analyst choices and cannot be audited.
Exceptions
Early exploration needs freedom; mark it as exploration rather than evidence.
Trade-offs
A rigid plan can miss unexpected findings, so allow documented amendments.
Pitfalls
Editing the plan after the results with no change log.
Lessons and concepts
Claims and their sources
Nosek and colleagues argue that defining research questions and the analysis plan before observing outcomes (preregistration) helps separate prediction from postdiction, and that strategies exist for pre-existing data.