CLAIM C28 of entrylib:2026.000008 Evaluating forecasts and classifiers: which measures, scoring rules and tests to use licence CC-BY-4.0; contributions C28 extensions review 0.1 DEFINITIONS D2 [concept] proper scoring rule: A scoring rule for probabilistic forecasts is proper when its expected value is best when the forecast distribution equals the distribution of the outcome, and strictly proper when it is best only then. [source B3] D5 [variable] RMSE: Root mean squared error: the square root of the mean of the squared differences between outcomes and point forecasts. [source B6] D7 [variable] MAE: Mean absolute error: the mean of the absolute differences between outcomes and point forecasts. [source B6] D13 [variable] interval score: The score of a central (1 - alpha) prediction interval: its width plus 2/alpha times the distance by which the outcome falls outside it. [source B3] CLAIMS C28 [methodological; principal; synthesis; support: argument W28; uses C49, C35, C44; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] When prediction intervals are compared over series on different scales, each series' [[D13:interval score]] should be scaled by a quantity of that series. scope: data_type: prediction intervals over series on different scales C35 [methodological; supporting; synthesis; support: cited B3, at Section 1 (Introduction), cited B3, at Abstract; limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B3", "locator": "Section 1 (Introduction)", "lab_event": "XT8", "exact": "a scoring rule for prediction intervals that is proper", "prefix": " the general idea dates back at\n", "suffix": " and has in- least to Brier ", "start": 15726, "end": 15780, "artefact": "A2"}, {"reference": "B3", "locator": "Abstract", "lab_event": "XT9", "exact": "propose the intuitively appealing interval score as a utility function in interval estimation that addresses width as well as\n coverage", "prefix": "tile\n estimation, and ", "suffix": ".\n KEY WORDS: Bayes fa", "start": 2499, "end": 2644, "artefact": "A2"}]; kind summary] The [[D13:interval score]] is a [[D2:proper scoring rule]] for central prediction intervals that addresses both their width and their coverage. scope: data_type: central prediction intervals C44 [methodological; supporting; synthesis; support: cited B6, at Section 2.1 (Scale-dependent measures), cited B6, at Section 2.4 (Relative measures); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B6", "locator": "Section 2.1 (Scale-dependent measures)", "lab_event": "XT17", "exact": "should\nnot be used, for example, when comparing across data sets that have different scales", "prefix": "ds on the same set of data, but ", "suffix": ". Never-\ntheless, the MSE was us", "start": 13183, "end": 13274, "artefact": "A7"}, {"reference": "B6", "locator": "Section 2.4 (Relative measures)", "lab_event": "XT18", "exact": "It makes no sense to compute the MAE across series (due to their different scales)", "prefix": "horizon across multiple\nseries. ", "suffix": ".\n\nA related approach is to use ", "start": 21483, "end": 21565, "artefact": "A7"}]; kind summary] Scale-dependent error measures, such as [[D5:RMSE]] and [[D7:MAE]], cannot be compared across series on different scales. scope: data_type: error measures over series on different scales C49 [methodological; supporting; synthesis; support: cited B18; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: kind summary] The M4 competition scored prediction intervals with the mean scaled interval score: the [[D13:interval score]] of each series scaled by an in-sample error of that series. scope: data_type: prediction intervals in the M4 competition WARRANTS W28 [deductive; premises C49, C35, C44] Interval scores of series on different scales cannot be compared unscaled; dividing each series' interval score by a positive quantity of that series that does not depend on the forecasts keeps it proper, as the mean scaled interval score of the M4 competition does. LIMITATIONS L3 [measurement; affects C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C44, C46; reduces_precision; mitigation none] Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions. L5 [researcher_position; affects C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C40, C41, C42, C43, C44, C45, C46, C47, C48, C49, C50, C51, C52, C53; could_bias_magnitude; mitigation none] The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. L6 [data_quality; affects C30, C33, C40, C41, C45, C47, C48, C49, C52, C53; reduces_precision; mitigation none] These claims rest in part on works read in full by the guarantor, not by the agent that wrote the claimset (lab note NT14); no passage of those works is quoted, so a reader checks these claims against the works themselves. ARTEFACTS A2 [data] Strictly proper scoring rules, prediction, and estimation (Gneiting and Raftery 2007, JASA 102(477), 359-378, doi:10.1198/016214506000001437), author's copy (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR2 on 2026-11-13. venue: Journal of the American Statistical Association. Written by others. A7 [data] Another look at measures of forecast accuracy (preprint of Hyndman and Koehler 2006, IJF 22(4), 679-688, doi:10.1016/j.ijforecast.2006.03.001) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR7 on 2026-11-13. venue: International Journal of Forecasting. Written by others. REFERENCES B3 10.1198/016214506000001437 (doi) B6 10.1016/j.ijforecast.2006.03.001 (doi) B18 10.1016/j.ijforecast.2019.04.014 (doi) HOW TO CITE This claim: Svetunkov, I. (guarantor) (2026). Claim C28, in: Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2#C28. https://entrylib.org/p/2026.000008/C28 The claimset it is from: Svetunkov, I. (guarantor) (2026). Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2. https://entrylib.org/p/2026.000008@v2