CLAIM C9 of entrylib:2026.000008 Evaluating forecasts and classifiers: which measures, scoring rules and tests to use licence CC-BY-4.0; contributions C9 extensions review 0.1 DEFINITIONS D1 [concept] consistent scoring function: A scoring function (an error measure for point forecasts) is consistent for a functional of the predictive distribution, such as the mean, the median or a quantile, when its expected value is lowest when the point forecast equals that functional. [source B2] D12 [variable] pinball loss: The quantile (asymmetric piecewise linear) loss at level tau: tau times the error when the outcome is above the forecast, and (1 - tau) times the absolute error when it is below. [source B2] CLAIMS C9 [methodological; principal; synthesis; support: argument W9; uses C34; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] Quantile forecasts should be evaluated with the [[D12:pinball loss]] at the quantile's level, averaged over the levels when there are several. scope: data_type: quantile forecasts C34 [methodological; supporting; synthesis; support: cited B2, at Abstract; Theorem 3.3 (arXiv:0912.0902 numbering); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B2", "locator": "Abstract; Theorem 3.3 (arXiv:0912.0902 numbering)", "lab_event": "XT7", "exact": "It is consistent for a quantile if and only if it is generalized piecewise linear", "prefix": " function. ", "suffix": ".\n ", "start": 3351, "end": 3432, "artefact": "A1"}]; kind summary] The [[D12:pinball loss]] at a level is a [[D1:consistent scoring function]] for the quantile at that level. scope: data_type: quantile forecasts WARRANTS W9 [deductive; premises C34] The pinball loss at a level is consistent for the quantile at that level, so its expected value is lowest for the correct quantile; averaging over the levels gives each quantile the same weight. LIMITATIONS L3 [measurement; affects C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C44, C46; reduces_precision; mitigation none] Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions. L5 [researcher_position; affects C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C40, C41, C42, C43, C44, C45, C46, C47, C48, C49, C50, C51, C52, C53; could_bias_magnitude; mitigation none] The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. ARTEFACTS A1 [data] Making and evaluating point forecasts (arXiv:0912.0902, the preprint of Gneiting 2011, JASA 106(494), 746-762, doi:10.1198/jasa.2011.r10138) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR1 on 2026-11-13. venue: arXiv. Written by others. REFERENCES B2 10.1198/jasa.2011.r10138 (doi) HOW TO CITE This claim: Svetunkov, I. (guarantor) (2026). Claim C9, in: Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2#C9. https://entrylib.org/p/2026.000008/C9 The claimset it is from: Svetunkov, I. (guarantor) (2026). Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2. https://entrylib.org/p/2026.000008@v2