CLAIM C27 of entrylib:2026.000008 Evaluating forecasts and classifiers: which measures, scoring rules and tests to use licence CC-BY-4.0; contributions C27 extensions review 0.1 DEFINITIONS D1 [concept] consistent scoring function: A scoring function (an error measure for point forecasts) is consistent for a functional of the predictive distribution, such as the mean, the median or a quantile, when its expected value is lowest when the point forecast equals that functional. [source B2] D5 [variable] RMSE: Root mean squared error: the square root of the mean of the squared differences between outcomes and point forecasts. [source B6] D7 [variable] MAE: Mean absolute error: the mean of the absolute differences between outcomes and point forecasts. [source B6] D12 [variable] pinball loss: The quantile (asymmetric piecewise linear) loss at level tau: tau times the error when the outcome is above the forecast, and (1 - tau) times the absolute error when it is below. [source B2] CLAIMS C27 [methodological; principal; synthesis; support: argument W27; uses C48, C34, C44; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] When quantile forecasts are compared over series on different scales, each series' [[D12:pinball loss]] should be scaled by a quantity of that series. scope: data_type: quantile forecasts over series on different scales C34 [methodological; supporting; synthesis; support: cited B2, at Abstract; Theorem 3.3 (arXiv:0912.0902 numbering); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B2", "locator": "Abstract; Theorem 3.3 (arXiv:0912.0902 numbering)", "lab_event": "XT7", "exact": "It is consistent for a quantile if and only if it is generalized piecewise linear", "prefix": " function. ", "suffix": ".\n ", "start": 3351, "end": 3432, "artefact": "A1"}]; kind summary] The [[D12:pinball loss]] at a level is a [[D1:consistent scoring function]] for the quantile at that level. scope: data_type: quantile forecasts C44 [methodological; supporting; synthesis; support: cited B6, at Section 2.1 (Scale-dependent measures), cited B6, at Section 2.4 (Relative measures); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B6", "locator": "Section 2.1 (Scale-dependent measures)", "lab_event": "XT17", "exact": "should\nnot be used, for example, when comparing across data sets that have different scales", "prefix": "ds on the same set of data, but ", "suffix": ". Never-\ntheless, the MSE was us", "start": 13183, "end": 13274, "artefact": "A7"}, {"reference": "B6", "locator": "Section 2.4 (Relative measures)", "lab_event": "XT18", "exact": "It makes no sense to compute the MAE across series (due to their different scales)", "prefix": "horizon across multiple\nseries. ", "suffix": ".\n\nA related approach is to use ", "start": 21483, "end": 21565, "artefact": "A7"}]; kind summary] Scale-dependent error measures, such as [[D5:RMSE]] and [[D7:MAE]], cannot be compared across series on different scales. scope: data_type: error measures over series on different scales C48 [methodological; supporting; synthesis; support: cited B1, at Probabilistic forecasts, cited B10; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B1", "locator": "Probabilistic forecasts", "lab_event": "XT22", "exact": "The precision of the probabilistic forecasts will be evaluated using the Scaled Pinball Loss (SPL) function", "prefix": "ition.\n\nProbabilistic forecasts\n", "suffix": ".\nThe measure is calculated for ", "start": 17832, "end": 17939, "artefact": "A9"}]; kind summary] The M5 uncertainty competition scored quantile forecasts with the scaled pinball loss: each series' [[D12:pinball loss]] divided by the in-sample mean absolute error of its one-step naive forecast. scope: data_type: quantile forecasts in the M5 uncertainty competition WARRANTS W27 [deductive; premises C48, C34, C44] Pinball losses of series on different scales cannot be compared unscaled; dividing each series' pinball loss by a quantity of that series that does not depend on the forecasts keeps the quantile it is consistent for, as the scaled pinball loss of the M5 uncertainty competition does. LIMITATIONS L3 [measurement; affects C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C44, C46; reduces_precision; mitigation none] Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions. L5 [researcher_position; affects C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C40, C41, C42, C43, C44, C45, C46, C47, C48, C49, C50, C51, C52, C53; could_bias_magnitude; mitigation none] The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. L6 [data_quality; affects C30, C33, C40, C41, C45, C47, C48, C49, C52, C53; reduces_precision; mitigation none] These claims rest in part on works read in full by the guarantor, not by the agent that wrote the claimset (lab note NT14); no passage of those works is quoted, so a reader checks these claims against the works themselves. ARTEFACTS A1 [data] Making and evaluating point forecasts (arXiv:0912.0902, the preprint of Gneiting 2011, JASA 106(494), 746-762, doi:10.1198/jasa.2011.r10138) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR1 on 2026-11-13. venue: arXiv. Written by others. A7 [data] Another look at measures of forecast accuracy (preprint of Hyndman and Koehler 2006, IJF 22(4), 679-688, doi:10.1016/j.ijforecast.2006.03.001) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR7 on 2026-11-13. venue: International Journal of Forecasting. Written by others. A9 [data] The M5 Competition: Competitors' Guide (the organisers' definition of the RMSSE, the scaled pinball loss and the reasons for them) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR9 on 2026-11-13. venue: M Open Forecasting Center, GitHub repository Mcompetitions/M5-methods. Written by others. REFERENCES B1 https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf (url) B2 10.1198/jasa.2011.r10138 (doi) B6 10.1016/j.ijforecast.2006.03.001 (doi) B10 10.1016/j.ijforecast.2021.10.009 (doi) HOW TO CITE This claim: Svetunkov, I. (guarantor) (2026). Claim C27, in: Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2#C27. https://entrylib.org/p/2026.000008/C27 The claimset it is from: Svetunkov, I. (guarantor) (2026). Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2. https://entrylib.org/p/2026.000008@v2