The claim
When point forecasts of the mean are compared over series on different scales, each series' RMSE should be scaled by the in-sample RMSE of its one-step naive forecast, for example, as in RMSSE.
Scope: data: point forecasts of the mean, over series on different scales · Argument: W3 · Uses: C47, C29, C46 · Limitations: L5
Claims it rests on
Squared error is a consistent scoring function for the mean.
Scope: data: point forecasts · Cited: Gneiting (2011), Section 5 (Discussion); Theorem 3.1 (arXiv:0912.0902 numbering) · Limitations: L3, L5 · Used by: C3
Scaled errors, as in MASE, are independent of the scale of the data and can be averaged over series on different scales.
Scope: data: error measures over series on different scales · Cited: Hyndman and Koehler (2006), Abstract; Section 3 (Scaled errors) · Limitations: L3, L5 · Used by: C3
The M5 accuracy competition scored point forecasts with the RMSSE: each series' squared errors divided by the in-sample mean squared error of its one-step naive forecast.
Scope: data: point forecasts in the M5 accuracy competition · Cited: https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf, Point forecasts · Cited: Makridakis et al. (2022) · Limitations: L5, L6 · Used by: C3
Evidence
Dividing a series' squared errors by a fixed positive number, such as the in-sample mean squared error of its one-step naive forecast, does not change which forecast minimises their expected value, so the scaled RMSE (the RMSSE) keeps the target of the RMSE, the mean, and the scaled values of different series can be averaged, as in the M5 accuracy competition.
Limitations
Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions.
The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it.
These claims rest in part on works read in full by the guarantor, not by the agent that wrote the claimset (lab note NT14); no passage of those works is quoted, so a reader checks these claims against the works themselves.
Definitions
Variables and measures 3
D5 RMSE: Root mean squared error: the square root of the mean of the squared differences between outcomes and point forecasts. Source: Hyndman and Koehler (2006).
D6 RMSSE: Root mean squared scaled error: the RMSE of a series' forecasts divided by the in-sample RMSE of the one-step naive forecast of that series, so that series on different scales can be averaged. Source: Makridakis et al. (2022).
D8 MASE: Mean absolute scaled error: the MAE of a series' forecasts divided by the in-sample MAE of the one-step naive forecast of that series. Source: Hyndman and Koehler (2006).
Concepts 2
D1 consistent scoring function: A scoring function (an error measure for point forecasts) is consistent for a functional of the predictive distribution, such as the mean, the median or a quantile, when its expected value is lowest when the point forecast equals that functional. Source: Gneiting (2011).
D3 point forecast of the mean: A single-valued forecast meant to equal the mean (expected value) of the outcome's distribution given the information available. Source: Gneiting (2011).
References
- Gneiting, T., 2011. Making and Evaluating Point Forecasts. Journal of the American Statistical Association 106, 746–762. https://doi.org/10.1198/jasa.2011.r10138 Used by: C29, D1, D3, A1
- Hyndman, R. J., Koehler, A. B., 2006. Another look at measures of forecast accuracy. International Journal of Forecasting 22, 679–688. https://doi.org/10.1016/j.ijforecast.2006.03.001 Used by: C46, D5, D8, A7
- Makridakis, S., Spiliotis, E., Assimakopoulos, V., 2022. M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting 38, 1346–1364. https://doi.org/10.1016/j.ijforecast.2021.11.013 Used by: C47, D6
- https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf Used by: C47, A9
Artefacts
- A1 · data · Gneiting (2011) Making and evaluating point forecasts (arXiv:0912.0902, the preprint of Gneiting 2011, JASA 106(494), 746-762, doi:10.1198/jasa.2011.r10138) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR1 on 2026-11-13. venue: arXiv. Written by others. 0912.0902
- A7 · data · Hyndman and Koehler (2006) Another look at measures of forecast accuracy (preprint of Hyndman and Koehler 2006, IJF 22(4), 679-688, doi:10.1016/j.ijforecast.2006.03.001) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR7 on 2026-11-13. venue: International Journal of Forecasting. Written by others. mase.pdf
- A9 · data · https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf The M5 Competition: Competitors' Guide (the organisers' definition of the RMSSE, the scaled pinball loss and the reasons for them) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR9 on 2026-11-13. venue: M Open Forecasting Center, GitHub repository Mcompetitions/M5-methods. Written by others. M5-Competitors-Guide.pdf
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C3: 79, Partly supported
Novelty: summarises earlier work, by the authors
- −10 Evidence Rests on cited work only; cited work read in full, without the passage it rests on; an argument from other claims. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −6 Open problems Limitation L5 (could bias magnitude, not mitigated): The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −3 Plan The review's search for the literature: none is recorded; the claims were confirmed against works the authors knew. To remove it: Record the protocol, then the searches and the screening of what they found (claimset lab protocol, search, screen), and export them.
- −1 Open problems Limitation L3 (reduces precision, not mitigated): Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L6 (reduces precision, not mitigated): These claims rest in part on works read in full by the guarantor, not by the agent that wrote the claimset (lab note NT14); no passage of those works is quoted, so a reader checks these claims against the works themselves. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C29: 93, Well supported
Novelty: summarises earlier work, by the authors
- −6 Open problems Limitation L5 (could bias magnitude, not mitigated): The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L3 (reduces precision, not mitigated): Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C46: 93, Well supported
Novelty: summarises earlier work, by the authors
- −6 Open problems Limitation L5 (could bias magnitude, not mitigated): The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L3 (reduces precision, not mitigated): Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C47: 93, Well supported
Novelty: summarises earlier work, by the authors
- −6 Open problems Limitation L5 (could bias magnitude, not mitigated): The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L6 (reduces precision, not mitigated): These claims rest in part on works read in full by the guarantor, not by the agent that wrote the claimset (lab note NT14); no passage of those works is quoted, so a reader checks these claims against the works themselves. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.