CLAIM C20 of entrylib:2026.000008 Evaluating forecasts and classifiers: which measures, scoring rules and tests to use licence CC-BY-4.0; contributions C20 extensions review 0.1 DEFINITIONS D6 [variable] RMSSE: Root mean squared scaled error: the RMSE of a series' forecasts divided by the in-sample RMSE of the one-step naive forecast of that series, so that series on different scales can be averaged. [source B9] D8 [variable] MASE: Mean absolute scaled error: the MAE of a series' forecasts divided by the in-sample MAE of the one-step naive forecast of that series. [source B6] CLAIMS C20 [methodological; principal; synthesis; support: argument W20; uses C46, C50; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] When methods are compared over many series, their accuracy should be summarised by the mean, the median and the quartiles of a scaled measure across the series, such as [[D6:RMSSE]] or [[D8:MASE]]. scope: data_type: comparisons of methods over many series C46 [methodological; supporting; synthesis; support: cited B6, at Abstract; Section 3 (Scaled errors); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B6", "locator": "Abstract; Section 3 (Scaled errors)", "lab_event": "XT19", "exact": "we propose that the mean absolute scaled error become the standard measure for\ncomparing forecast accuracy across multiple time series", "prefix": "ccurring situ-\nations. Instead, ", "suffix": ".\n\n\n\nKeywords: forecast accuracy", "start": 840, "end": 974, "artefact": "A7"}]; kind summary] Scaled errors, as in [[D8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. scope: data_type: error measures over series on different scales C50 [methodological; supporting; synthesis; support: cited B5, at Section 3.1 (Comparisons of two classifiers); limitations L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B5", "locator": "Section 3.1 (Comparisons of two classifiers)", "lab_event": "XT23", "exact": "Averages are also susceptible to outliers", "prefix": "ms\nwith similar properties.\n ", "suffix": ". They allow classifier’s excell", "start": 18817, "end": 18858, "artefact": "A4"}]; kind summary] Averages of a performance measure over many data sets are susceptible to outliers. scope: data_type: comparisons of methods over many data sets WARRANTS W20 [deductive; premises C46, C50] A scaled measure can be averaged over series, but averages over many data sets are susceptible to outliers; the median and the quartiles show how accuracy is spread over the series, which the mean alone hides. LIMITATIONS L3 [measurement; affects C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C44, C46; reduces_precision; mitigation none] Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions. L5 [researcher_position; affects C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C40, C41, C42, C43, C44, C45, C46, C47, C48, C49, C50, C51, C52, C53; could_bias_magnitude; mitigation none] The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. ARTEFACTS A4 [data] Statistical comparisons of classifiers over multiple data sets (Demsar 2006, JMLR 7, 1-30) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR4 on 2026-11-13. venue: Journal of Machine Learning Research. Written by others. A7 [data] Another look at measures of forecast accuracy (preprint of Hyndman and Koehler 2006, IJF 22(4), 679-688, doi:10.1016/j.ijforecast.2006.03.001) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR7 on 2026-11-13. venue: International Journal of Forecasting. Written by others. REFERENCES B5 https://jmlr.org/papers/v7/demsar06a.html (url) B6 10.1016/j.ijforecast.2006.03.001 (doi) B9 10.1016/j.ijforecast.2021.11.013 (doi) HOW TO CITE This claim: Svetunkov, I. (guarantor) (2026). Claim C20, in: Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2#C20. https://entrylib.org/p/2026.000008/C20 The claimset it is from: Svetunkov, I. (guarantor) (2026). Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2. https://entrylib.org/p/2026.000008@v2