CLAIM C26 of entrylib:2026.000008 Evaluating forecasts and classifiers: which measures, scoring rules and tests to use licence CC-BY-4.0; contributions C26 extensions review 0.1 DEFINITIONS D5 [variable] RMSE: Root mean squared error: the square root of the mean of the squared differences between outcomes and point forecasts. [source B6] D6 [variable] RMSSE: Root mean squared scaled error: the RMSE of a series' forecasts divided by the in-sample RMSE of the one-step naive forecast of that series, so that series on different scales can be averaged. [source B9] D7 [variable] MAE: Mean absolute error: the mean of the absolute differences between outcomes and point forecasts. [source B6] D8 [variable] MASE: Mean absolute scaled error: the MAE of a series' forecasts divided by the in-sample MAE of the one-step naive forecast of that series. [source B6] CLAIMS C26 [methodological; principal; synthesis; support: argument W26; uses C44, C46, C47; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] To aggregate the accuracy of point forecasts over series on different scales, each series' errors should first be scaled by a quantity of that series that does not depend on the compared forecasts, such as the in-sample error of its one-step naive forecast in [[D8:MASE]] or [[D6:RMSSE]], before the scaled errors are averaged. scope: data_type: point forecasts over series on different scales C44 [methodological; supporting; synthesis; support: cited B6, at Section 2.1 (Scale-dependent measures), cited B6, at Section 2.4 (Relative measures); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B6", "locator": "Section 2.1 (Scale-dependent measures)", "lab_event": "XT17", "exact": "should\nnot be used, for example, when comparing across data sets that have different scales", "prefix": "ds on the same set of data, but ", "suffix": ". Never-\ntheless, the MSE was us", "start": 13183, "end": 13274, "artefact": "A7"}, {"reference": "B6", "locator": "Section 2.4 (Relative measures)", "lab_event": "XT18", "exact": "It makes no sense to compute the MAE across series (due to their different scales)", "prefix": "horizon across multiple\nseries. ", "suffix": ".\n\nA related approach is to use ", "start": 21483, "end": 21565, "artefact": "A7"}]; kind summary] Scale-dependent error measures, such as [[D5:RMSE]] and [[D7:MAE]], cannot be compared across series on different scales. scope: data_type: error measures over series on different scales C46 [methodological; supporting; synthesis; support: cited B6, at Abstract; Section 3 (Scaled errors); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B6", "locator": "Abstract; Section 3 (Scaled errors)", "lab_event": "XT19", "exact": "we propose that the mean absolute scaled error become the standard measure for\ncomparing forecast accuracy across multiple time series", "prefix": "ccurring situ-\nations. Instead, ", "suffix": ".\n\n\n\nKeywords: forecast accuracy", "start": 840, "end": 974, "artefact": "A7"}]; kind summary] Scaled errors, as in [[D8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. scope: data_type: error measures over series on different scales C47 [methodological; supporting; synthesis; support: cited B1, at Point forecasts, cited B9; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B1", "locator": "Point forecasts", "lab_event": "XT20", "exact": "The accuracy of the point forecasts will be evaluated using the Root Mean Squared Scaled Error (RMSSE)", "prefix": "ce-\nprogram\n\n5\n\fPoint forecasts\n", "suffix": ",\nwhich is a variant of the well", "start": 14467, "end": 14569, "artefact": "A9"}, {"reference": "B1", "locator": "Point forecasts", "lab_event": "XT21", "exact": "The measure is scale independent, meaning that it can be effectively used to compare forecasts\n across series with different scales", "prefix": "optimized for the mean.\n  ", "suffix": ".\n  In contrast to other m", "start": 16197, "end": 16335, "artefact": "A9"}]; kind summary] The M5 accuracy competition scored point forecasts with the [[D6:RMSSE]]: each series' squared errors divided by the in-sample mean squared error of its one-step naive forecast. scope: data_type: point forecasts in the M5 accuracy competition WARRANTS W26 [deductive; premises C44, C46, C47] Scaled errors are independent of the scale of the data and can be averaged over series; dividing by a quantity of the series that does not depend on the compared forecasts keeps the target that the unscaled measure is consistent for. LIMITATIONS L3 [measurement; affects C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C44, C46; reduces_precision; mitigation none] Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions. L5 [researcher_position; affects C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C40, C41, C42, C43, C44, C45, C46, C47, C48, C49, C50, C51, C52, C53; could_bias_magnitude; mitigation none] The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. L6 [data_quality; affects C30, C33, C40, C41, C45, C47, C48, C49, C52, C53; reduces_precision; mitigation none] These claims rest in part on works read in full by the guarantor, not by the agent that wrote the claimset (lab note NT14); no passage of those works is quoted, so a reader checks these claims against the works themselves. ARTEFACTS A7 [data] Another look at measures of forecast accuracy (preprint of Hyndman and Koehler 2006, IJF 22(4), 679-688, doi:10.1016/j.ijforecast.2006.03.001) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR7 on 2026-11-13. venue: International Journal of Forecasting. Written by others. A9 [data] The M5 Competition: Competitors' Guide (the organisers' definition of the RMSSE, the scaled pinball loss and the reasons for them) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR9 on 2026-11-13. venue: M Open Forecasting Center, GitHub repository Mcompetitions/M5-methods. Written by others. REFERENCES B1 https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf (url) B6 10.1016/j.ijforecast.2006.03.001 (doi) B9 10.1016/j.ijforecast.2021.11.013 (doi) HOW TO CITE This claim: Svetunkov, I. (guarantor) (2026). Claim C26, in: Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2#C26. https://entrylib.org/p/2026.000008/C26 The claimset it is from: Svetunkov, I. (guarantor) (2026). Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2. https://entrylib.org/p/2026.000008@v2