How do you validate a marketing forecast?
A forecast is validated on periods not used for fitting, with a realistic forecast origin, a naive benchmark and a metric aligned with error cost.
Direct answer
Compare models for a declared horizon and error cost.
A forecast is validated on periods not used for fitting, with a realistic forecast origin, a naive benchmark and a metric aligned with error cost.
MASE=[Σtest|yt−ŷt|/H] / [Σtrain|yt−yt−m|/(T−m)]ESTIMAND
Estimation profile
- Unit of analysis
- Observation, entity or series × periodControlled scientific terminology
series × forecast origin × horizon - Exact estimand
Out-of-sample point error and interval calibration- Model or deliverable
MASE=[Σtest|yt−ŷt|/H] / [Σtrain|yt−yt−m|/(T−m)]
01—08
Verifiable analysis framework
- 01Question
How do you validate a marketing forecast?
- 02Estimand
Out-of-sample point error and interval calibration · series × forecast origin × horizon
- 03Data
date · outcome · forecast_origin · horizon · prediction · interval
- 04Model or deliverable
MASE=[Σtest|yt−ŷt|/H] / [Σtrain|yt−yt−m|/(T−m)]
- 05Declared calculation
MASE=[Σtest|yt−ŷt|/H] / [Σtrain|yt−yt−m|/(T−m)] → Out-of-sample point error and interval calibration
- 06Uncertainty checks
Rolling origin · Naive benchmark · Non-zero scaling denominator · Interval coverage and score · Data revisions
- 07Method-specific validation
Check data, estimate stability and interpretation limits. Rolling origin · Naive benchmark · Non-zero scaling denominator · Interval coverage and score · Data revisions
- 08Limitations
Guarantee future performance after a regime change.
Required variables
dateoutcomeforecast_originhorizonpredictioninterval
Checks · Controlled scientific terminology
- Rolling origin
- Naive benchmark
- Non-zero scaling denominator
- Interval coverage and score
- Data revisions
Compare models for a declared horizon and error cost.
Guarantee future performance after a regime change.
Scientific sources
2 sources
Method connections