CLAIMSET Evaluating forecasts and classifiers: which measures, scoring rules and tests to use summary How should a study measure and compare predictions? This review states what the methodological literature establishes and the rules it draws from it. A measure has to match what a prediction targets: squared error (RMSE) for forecasts of the mean, absolute error (MAE) for the median, the pinball loss for a quantile, the interval score with coverage for intervals, CRPS or the log score for distributions, and log loss or the Brier score for class probabilities. MAPE rewards forecasts that are too low, sMAPE prefers forecasts that are too high, and AUC and accuracy do not judge probabilities. Errors of series on different scales should be scaled per series before they are averaged, never summed as they are. The mean, median and quartiles of a scaled measure summarise accuracy over many series; rank-based tests (Friedman with Nemenyi, MCB) and the Diebold-Mariano test tell whether similar methods differ statistically. licence CC-BY-4.0; contributions C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28; highlights C1, C26, C5, C12, C15 declarations funding: none | competing interests: Ivan Svetunkov runs EntryLib, which publishes the study-planning guides that this claimset summarises. extensions review 0.1 [review: flow {"assessed": 20, "excluded_full_text": {"out of scope: it was cited only for implementation facts of ETS models and back-transformed forecasts, which decision DC1 left out of this study": 2}, "included": 18, "other_sources": 18}] exploration 1 question; hypotheses 55: 53 supported, 2 abandoned; tests 0; runs 0; lab record A10 DEFINITIONS D1 [concept] consistent scoring function: A scoring function (an error measure for point forecasts) is consistent for a functional of the predictive distribution, such as the mean, the median or a quantile, when its expected value is lowest when the point forecast equals that functional. [source B2] D2 [concept] proper scoring rule: A scoring rule for probabilistic forecasts is proper when its expected value is best when the forecast distribution equals the distribution of the outcome, and strictly proper when it is best only then. [source B3] D3 [concept] point forecast of the mean: A single-valued forecast meant to equal the mean (expected value) of the outcome's distribution given the information available. [source B2] D4 [concept] point forecast of the median: A single-valued forecast meant to equal the median of the outcome's distribution given the information available. [source B2] D5 [variable] RMSE: Root mean squared error: the square root of the mean of the squared differences between outcomes and point forecasts. [source B6] D6 [variable] RMSSE: Root mean squared scaled error: the RMSE of a series' forecasts divided by the in-sample RMSE of the one-step naive forecast of that series, so that series on different scales can be averaged. [source B9] D7 [variable] MAE: Mean absolute error: the mean of the absolute differences between outcomes and point forecasts. [source B6] D8 [variable] MASE: Mean absolute scaled error: the MAE of a series' forecasts divided by the in-sample MAE of the one-step naive forecast of that series. [source B6] D9 [variable] MAPE: Mean absolute percentage error: the mean of the absolute differences between outcomes and point forecasts, each divided by the outcome. [source B6] D10 [variable] sMAPE: Symmetric mean absolute percentage error: the mean of the absolute differences between outcomes and point forecasts, each divided by the mean of the absolute outcome and the absolute forecast. [source B11] D11 [concept] median of order -1: The beta-median with beta = -1: the point forecast that minimises the expected absolute percentage error; for positive outcomes it lies below the median. [source B2] D12 [variable] pinball loss: The quantile (asymmetric piecewise linear) loss at level tau: tau times the error when the outcome is above the forecast, and (1 - tau) times the absolute error when it is below. [source B2] D13 [variable] interval score: The score of a central (1 - alpha) prediction interval: its width plus 2/alpha times the distance by which the outcome falls outside it. [source B3] D14 [variable] coverage: The share of outcomes that fall inside the prediction intervals, compared with their nominal level. [source B3] D15 [variable] CRPS: Continuous ranked probability score: the integral over thresholds of the squared difference between the forecast's cumulative distribution function and the step function at the outcome. [source B3] D16 [variable] log score: The logarithmic score: the negative logarithm of the forecast density (or probability) at the outcome. [source B3] D17 [variable] Brier score: The mean squared difference between forecast probabilities of a class and the outcome coded 1 if the class occurred and 0 otherwise. [source B13] D18 [concept] probability integral transform: The value of the forecast's cumulative distribution function at the outcome; for a calibrated forecast its values are uniformly distributed. [source B4] D19 [concept] calibration: Statistical consistency between forecast distributions and outcomes: events forecast with a given probability occur with that frequency. [source B4] D20 [concept] sharpness: The concentration of the forecast distributions, a property of the forecasts alone; among calibrated forecasts, sharper ones are better. [source B4] D21 [variable] accuracy: The share of cases whose predicted class, after thresholding the predicted probabilities, equals the observed class. D22 [variable] AUC: The area under the ROC curve: the probability that a randomly chosen case of the positive class receives a higher score than a randomly chosen case of the negative class. [source B14] D23 [concept] precision-recall curve: The curve of precision (the share of predicted positives that are positive) against recall (the share of positives predicted as positive) over the classification thresholds. [source B7] D24 [data] intermittent series: A time series with many zero values, such as the sales of a slow-moving item. [source B12] D25 [method] Friedman test: A non-parametric test of whether several methods differ, applied to their ranks on each of several data sets. [source B5] D26 [method] Nemenyi test: A post-hoc test after the Friedman test that compares every pair of methods by their mean ranks against a critical difference. [source B5] D27 [method] MCB: Multiple comparisons with the best: a test, on mean ranks over series, of which methods differ significantly from the best-ranked one. [source B15] D28 [method] Diebold-Mariano test: A test of equal expected loss of two competing forecasts of the same series, based on the mean loss differential and its long-run variance. [source B16] CLAIMS C1 [methodological; principal; synthesis; support: argument W1; uses C30; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] Point forecasts should be evaluated with a [[D1:consistent scoring function]] for the functional they target, such as the mean, the median or a quantile. scope: data_type: point forecasts C2 [methodological; principal; synthesis; support: argument W2; uses C29, C30; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] [[D3:Point forecasts of the mean]] should be evaluated with squared error, as in [[D5:RMSE]]. scope: data_type: point forecasts of the mean C3 [methodological; principal; synthesis; support: argument W3; uses C47, C29, C46; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] When [[D3:point forecasts of the mean]] are compared over series on different scales, each series' [[D5:RMSE]] should be scaled by the in-sample RMSE of its one-step naive forecast, for example, as in [[D6:RMSSE]]. scope: data_type: point forecasts of the mean, over series on different scales C4 [methodological; principal; synthesis; support: argument W4; uses C31, C29; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] [[D7:MAE]] and [[D8:MASE]] can be used as the main measure of point forecasts only when the mean and median forecasts coincide. scope: data_type: point forecasts of the mean C5 [methodological; principal; synthesis; support: argument W5; uses C32; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] [[D9:MAPE]] should not be the measure for point forecasts of the mean or the median. scope: data_type: point forecasts of positive outcomes C6 [methodological; principal; synthesis; support: argument W6; uses C33; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] [[D10:sMAPE]] should not be the measure for point forecasts. scope: data_type: point forecasts C7 [methodological; principal; synthesis; support: argument W7; uses C31, C46; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] [[D4:Point forecasts of the median]] should be evaluated with absolute error, as in [[D7:MAE]] or, across series on different scales, [[D8:MASE]]. scope: data_type: point forecasts of the median C8 [methodological; principal; synthesis; support: argument W8; uses C41, C31; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] On [[D24:intermittent series]], absolute-error measures should be avoided. scope: data_type: point forecasts of intermittent series C9 [methodological; principal; synthesis; support: argument W9; uses C34; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] Quantile forecasts should be evaluated with the [[D12:pinball loss]] at the quantile's level, averaged over the levels when there are several. scope: data_type: quantile forecasts C10 [methodological; principal; synthesis; support: argument W10; uses C35; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] Prediction intervals should be evaluated with the [[D13:interval score]], reported together with their [[D14:coverage]] against the nominal level. scope: data_type: central prediction intervals C11 [methodological; principal; synthesis; support: argument W11; uses C35; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] [[D14:Coverage]] alone should not be the measure of prediction intervals. scope: data_type: prediction intervals C12 [methodological; principal; synthesis; support: argument W12; uses C36, C37; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] Predictive distributions should be evaluated with a [[D2:proper scoring rule]], such as the [[D15:CRPS]] or the [[D16:log score]]. scope: data_type: predictive distributions C13 [methodological; principal; synthesis; support: argument W13; uses C39, C35; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] Predictive distributions and prediction intervals should first be checked for [[D19:calibration]], and their [[D20:sharpness]] compared only among forecasts that are calibrated. scope: data_type: predictive distributions and prediction intervals C14 [methodological; principal; synthesis; support: argument W14; uses C40, C36, C37; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] Predictive distributions of counts, such as forecasts of intermittent sales, should be evaluated with proper scoring rules and a randomised [[D18:probability integral transform]]. scope: data_type: predictive distributions of counts C15 [methodological; principal; synthesis; support: argument W15; uses C38, C37; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] Forecast probabilities of classes should be evaluated with a [[D2:proper scoring rule]], such as the [[D16:log score]] or the [[D17:Brier score]]. scope: data_type: forecast probabilities of classes C16 [methodological; principal; extension; support: argument W16; uses C38, C37; limitations L4, L5; strength moderate (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] [[D21:Accuracy]] should not be the measure of forecast class probabilities. scope: data_type: forecast class probabilities, thresholded to labels extends B3: Gneiting and Raftery (2007) show that the Brier and logarithmic scores are proper for forecast probabilities of classes. C17 [methodological; principal; synthesis; support: argument W17; uses C42; limitations L1, L5; strength moderate (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] [[D22:AUC]] should not be the measure of forecast class probabilities. scope: data_type: forecast class probabilities, ranked by a classifier's scores C18 [methodological; principal; synthesis; support: argument W18; uses C43; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] With rare positive cases, the [[D23:precision-recall curve]] should be reported rather than the ROC curve alone. scope: data_type: binary classifiers | conditions: imbalanced classes C19 [methodological; principal; synthesis; support: argument W19; uses C30; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] When the compared methods target different functionals, one the mean and another the median, each should be evaluated with a measure consistent for its own target, or the methods made to target the same functional. scope: data_type: point forecasts of methods that target different functionals C20 [methodological; principal; synthesis; support: argument W20; uses C46, C50; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] When methods are compared over many series, their accuracy should be summarised by the mean, the median and the quartiles of a scaled measure across the series, such as [[D6:RMSSE]] or [[D8:MASE]]. scope: data_type: comparisons of methods over many series C21 [methodological; principal; synthesis; support: argument W21; uses C51; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] To tell whether several methods whose aggregate accuracy is similar differ statistically significantly over many series or data sets, their ranks on each series should be tested with the [[D25:Friedman test]] followed by [[D26:Nemenyi test]] post-hoc comparisons. scope: data_type: comparisons of several methods over many series or data sets C22 [methodological; principal; synthesis; support: argument W22; uses C52; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] To tell which methods differ statistically significantly from the best one over many series, multiple comparisons with the best ([[D27:MCB]]) on ranks should be used. scope: data_type: comparisons of methods over many series C23 [methodological; principal; synthesis; support: argument W23; uses C53; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] The accuracy of two forecasts of the same series should be compared with the [[D28:Diebold-Mariano test]]. scope: data_type: two forecasts of one series C24 [methodological; principal; synthesis; support: argument W24; uses C30; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] An error measure should be chosen to match what the compared methods predict, even if competitions or well-known studies used a different one. scope: data_type: evaluations of point forecasts C25 [methodological; principal; synthesis; support: argument W25; uses C44, C45; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] Scale-dependent error measures, such as [[D5:RMSE]] and [[D7:MAE]], should not be summed or averaged across series on different scales. scope: data_type: point forecasts over series on different scales C26 [methodological; principal; synthesis; support: argument W26; uses C44, C46, C47; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] To aggregate the accuracy of point forecasts over series on different scales, each series' errors should first be scaled by a quantity of that series that does not depend on the compared forecasts, such as the in-sample error of its one-step naive forecast in [[D8:MASE]] or [[D6:RMSSE]], before the scaled errors are averaged. scope: data_type: point forecasts over series on different scales C27 [methodological; principal; synthesis; support: argument W27; uses C48, C34, C44; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] When quantile forecasts are compared over series on different scales, each series' [[D12:pinball loss]] should be scaled by a quantity of that series. scope: data_type: quantile forecasts over series on different scales C28 [methodological; principal; synthesis; support: argument W28; uses C49, C35, C44; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] When prediction intervals are compared over series on different scales, each series' [[D13:interval score]] should be scaled by a quantity of that series. scope: data_type: prediction intervals over series on different scales C29 [methodological; supporting; synthesis; support: cited B2, at Section 5 (Discussion); Theorem 3.1 (arXiv:0912.0902 numbering); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B2", "locator": "Section 5 (Discussion); Theorem 3.1 (arXiv:0912.0902 numbering)", "lab_event": "XT1", "exact": "the squared error scoring function is consistent for the mean", "prefix": "businesses and organi-\nzations, ", "suffix": ", and the absolute error\nscoring", "start": 74674, "end": 74735, "artefact": "A1"}]; kind summary] Squared error is a [[D1:consistent scoring function]] for the mean. scope: data_type: point forecasts C30 [methodological; supporting; synthesis; support: cited B2, at Abstract (arXiv:0912.0902 numbering), cited B8; limitations L3, L5, L6; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B2", "locator": "Abstract (arXiv:0912.0902 numbering)", "lab_event": "XT2", "exact": "it is critical that the\n scoring function be consistent for it", "prefix": "ve in the form of a functional, ", "suffix": ", in the sense that the expected", "start": 2433, "end": 2535, "artefact": "A1"}]; kind summary] Which point forecast has the lowest expected score depends on the scoring function: a [[D1:consistent scoring function]] for a functional, such as the mean or a quantile, is lowest in expectation for the forecast that equals that functional of the outcome's distribution. scope: data_type: point forecasts C31 [methodological; supporting; synthesis; support: cited B2, at Section 5 (Discussion) (arXiv:0912.0902 numbering); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B2", "locator": "Section 5 (Discussion) (arXiv:0912.0902 numbering)", "lab_event": "XT3", "exact": "the absolute error\nscoring function for the median", "prefix": "is consistent for the mean, and ", "suffix": ". The absolute percentage error ", "start": 74741, "end": 74791, "artefact": "A1"}]; kind summary] Absolute error is a [[D1:consistent scoring function]] for the median. scope: data_type: point forecasts C32 [methodological; supporting; synthesis; support: cited B2, at Section 5 (Discussion); the beta-median in Section 1.3 (arXiv:0912.0902 numbering), cited B2, at Section 5 (Discussion) (arXiv:0912.0902 numbering); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B2", "locator": "Section 5 (Discussion); the beta-median in Section 1.3 (arXiv:0912.0902 numbering)", "lab_event": "XT4", "exact": "the median of order −1", "prefix": "on-standard functional, namely, ", "suffix": ", med(−1) , which tends to sup-\n", "start": 74980, "end": 75002, "artefact": "A1"}, {"reference": "B2", "locator": "Section 5 (Discussion) (arXiv:0912.0902 numbering)", "lab_event": "XT5", "exact": "severe underforecasts, as compared to the mean or median", "prefix": "(−1) , which tends to sup-\nport ", "suffix": ". It thus seems prudent that\nbus", "start": 75039, "end": 75095, "artefact": "A1"}]; kind summary] The absolute percentage error averaged by [[D9:MAPE]] is consistent for the [[D11:median of order -1]], a functional that supports severe underforecasts compared with the mean or the median. scope: data_type: point forecasts of positive outcomes C33 [methodological; supporting; synthesis; support: cited B6, at Section 2.2 (Measures based on percentage errors), cited B11; limitations L3, L5, L6; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B6", "locator": "Section 2.2 (Measures based on percentage errors)", "lab_event": "XT6", "exact": "has a\nheavier penalty when forecasts are low compared to when forecasts are high", "prefix": "value of 2|Yt − Ft |/(Yt + Ft ) ", "suffix": ". See Goodwin\nand Lawton (1999) ", "start": 17140, "end": 17220, "artefact": "A7"}]; kind summary] For the same outcome, [[D10:sMAPE]] penalises a forecast below the outcome more heavily than a forecast above it. scope: data_type: point forecasts of positive outcomes C34 [methodological; supporting; synthesis; support: cited B2, at Abstract; Theorem 3.3 (arXiv:0912.0902 numbering); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B2", "locator": "Abstract; Theorem 3.3 (arXiv:0912.0902 numbering)", "lab_event": "XT7", "exact": "It is consistent for a quantile if and only if it is generalized piecewise linear", "prefix": " function. ", "suffix": ".\n ", "start": 3351, "end": 3432, "artefact": "A1"}]; kind summary] The [[D12:pinball loss]] at a level is a [[D1:consistent scoring function]] for the quantile at that level. scope: data_type: quantile forecasts C35 [methodological; supporting; synthesis; support: cited B3, at Section 1 (Introduction), cited B3, at Abstract; limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B3", "locator": "Section 1 (Introduction)", "lab_event": "XT8", "exact": "a scoring rule for prediction intervals that is proper", "prefix": " the general idea dates back at\n", "suffix": " and has in- least to Brier ", "start": 15726, "end": 15780, "artefact": "A2"}, {"reference": "B3", "locator": "Abstract", "lab_event": "XT9", "exact": "propose the intuitively appealing interval score as a utility function in interval estimation that addresses width as well as\n coverage", "prefix": "tile\n estimation, and ", "suffix": ".\n KEY WORDS: Bayes fa", "start": 2499, "end": 2644, "artefact": "A2"}]; kind summary] The [[D13:interval score]] is a [[D2:proper scoring rule]] for central prediction intervals that addresses both their width and their coverage. scope: data_type: central prediction intervals C36 [methodological; supporting; synthesis; support: cited B3, at Section 4.2 (Continuous ranked probability score); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B3", "locator": "Section 4.2 (Continuous ranked probability score)", "lab_event": "XT10", "exact": "The CRPS is proper relative to the class P", "prefix": "003) implies the equality of\n ", "suffix": " and strictly proper\n ", "start": 64068, "end": 64110, "artefact": "A2"}]; kind summary] The [[D15:CRPS]] is a [[D2:proper scoring rule]] for predictive distributions of continuous outcomes. scope: data_type: predictive distributions of continuous outcomes C37 [methodological; supporting; synthesis; support: cited B3, at Section 4.1; limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B3", "locator": "Section 4.1", "lab_event": "XT11", "exact": "The logarithmic score is strictly proper", "prefix": "rithmic score (Ta- 2002). ", "suffix": " relative to the\nble 1), the asy", "start": 53590, "end": 53630, "artefact": "A2"}]; kind summary] The [[D16:log score]] is a strictly [[D2:proper scoring rule]]. scope: data_type: predictive distributions C38 [methodological; supporting; synthesis; support: cited B3, at Section 3, Example 1 (Quadratic or Brier score), cited B13; limitations L2, L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B3", "locator": "Section 3, Example 1 (Quadratic or Brier score)", "lab_event": "XT12", "exact": "Example 1 (Quadratic or Brier score)", "prefix": " \u0005\n ", "suffix": ". If G(p) = m j=1 pj −\n ", "start": 33270, "end": 33306, "artefact": "A2"}]; kind summary] The [[D17:Brier score]] is a [[D2:proper scoring rule]] for forecast probabilities of classes. scope: data_type: forecast probabilities of classes C39 [methodological; supporting; synthesis; support: cited B4, at Abstract; limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B4", "locator": "Abstract", "lab_event": "XT13", "exact": "is based on the paradigm of maximizing the sharpness of\n the predictive distributions subject to calibration", "prefix": " of predictive performance that ", "suffix": ". Calibration refers to the stat", "start": 635, "end": 748, "artefact": "A3"}]; kind summary] Predictive distributions are assessed by maximising their [[D20:sharpness]] subject to their [[D19:calibration]]. scope: data_type: predictive distributions C40 [methodological; supporting; synthesis; support: cited B12; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: kind summary] The calibration of predictive distributions of counts can be assessed with a randomised [[D18:probability integral transform]]. scope: data_type: predictive distributions of counts C41 [methodological; supporting; synthesis; support: cited B1, at Point forecasts, cited B12; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B1", "locator": "Point forecasts", "lab_event": "XT14", "exact": "absolute errors, which are optimized for the median, would assign lower scores\n (better performance) to forecasting methods that derive forecasts close to zero", "prefix": " zeros.\n This means that ", "suffix": ". However, the\n objective", "start": 15824, "end": 15990, "artefact": "A9"}]; kind summary] On [[D24:intermittent series]], absolute-error measures give their best scores to forecasts close to zero. scope: data_type: point forecasts of intermittent series C42 [methodological; supporting; synthesis; support: cited B14; limitations L1, L5; verifiability open; planning exploratory; outcome supported] [review: kind summary] The [[D22:AUC]] is equivalent to averaging the misclassification loss over a distribution of misclassification costs that depends on the classifier. scope: data_type: classifiers that score cases C43 [methodological; supporting; synthesis; support: cited B7, at Title, cited B7, at Abstract; limitations L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B7", "locator": "Title", "lab_event": "XT15", "exact": "The Precision-Recall Plot Is More Informative\n than the ROC Plot When Evaluating Binary\n Classifiers on Imbalanced Datasets", "prefix": " ", "suffix": "\n ", "start": 135, "end": 372, "artefact": "A8"}, {"reference": "B7", "locator": "Abstract", "lab_event": "XT16", "exact": "can be deceptive", "prefix": " context of imbalanced datasets ", "suffix": " with respect to conclu-\nROC Plo", "start": 2381, "end": 2397, "artefact": "A8"}]; kind summary] On imbalanced data, the [[D23:precision-recall curve]] is more informative than the ROC curve for evaluating binary classifiers. scope: data_type: binary classifiers on imbalanced data C44 [methodological; supporting; synthesis; support: cited B6, at Section 2.1 (Scale-dependent measures), cited B6, at Section 2.4 (Relative measures); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B6", "locator": "Section 2.1 (Scale-dependent measures)", "lab_event": "XT17", "exact": "should\nnot be used, for example, when comparing across data sets that have different scales", "prefix": "ds on the same set of data, but ", "suffix": ". Never-\ntheless, the MSE was us", "start": 13183, "end": 13274, "artefact": "A7"}, {"reference": "B6", "locator": "Section 2.4 (Relative measures)", "lab_event": "XT18", "exact": "It makes no sense to compute the MAE across series (due to their different scales)", "prefix": "horizon across multiple\nseries. ", "suffix": ".\n\nA related approach is to use ", "start": 21483, "end": 21565, "artefact": "A7"}]; kind summary] Scale-dependent error measures, such as [[D5:RMSE]] and [[D7:MAE]], cannot be compared across series on different scales. scope: data_type: error measures over series on different scales C45 [methodological; supporting; synthesis; support: cited B17; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: kind summary] When the errors of series on different scales are summed or averaged without scaling, the series with the largest values dominate the result. scope: data_type: error measures over series on different scales C46 [methodological; supporting; synthesis; support: cited B6, at Abstract; Section 3 (Scaled errors); limitations L3, L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B6", "locator": "Abstract; Section 3 (Scaled errors)", "lab_event": "XT19", "exact": "we propose that the mean absolute scaled error become the standard measure for\ncomparing forecast accuracy across multiple time series", "prefix": "ccurring situ-\nations. Instead, ", "suffix": ".\n\n\n\nKeywords: forecast accuracy", "start": 840, "end": 974, "artefact": "A7"}]; kind summary] Scaled errors, as in [[D8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. scope: data_type: error measures over series on different scales C47 [methodological; supporting; synthesis; support: cited B1, at Point forecasts, cited B9; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B1", "locator": "Point forecasts", "lab_event": "XT20", "exact": "The accuracy of the point forecasts will be evaluated using the Root Mean Squared Scaled Error (RMSSE)", "prefix": "ce-\nprogram\n\n5\n\fPoint forecasts\n", "suffix": ",\nwhich is a variant of the well", "start": 14467, "end": 14569, "artefact": "A9"}, {"reference": "B1", "locator": "Point forecasts", "lab_event": "XT21", "exact": "The measure is scale independent, meaning that it can be effectively used to compare forecasts\n across series with different scales", "prefix": "optimized for the mean.\n  ", "suffix": ".\n  In contrast to other m", "start": 16197, "end": 16335, "artefact": "A9"}]; kind summary] The M5 accuracy competition scored point forecasts with the [[D6:RMSSE]]: each series' squared errors divided by the in-sample mean squared error of its one-step naive forecast. scope: data_type: point forecasts in the M5 accuracy competition C48 [methodological; supporting; synthesis; support: cited B1, at Probabilistic forecasts, cited B10; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B1", "locator": "Probabilistic forecasts", "lab_event": "XT22", "exact": "The precision of the probabilistic forecasts will be evaluated using the Scaled Pinball Loss (SPL) function", "prefix": "ition.\n\nProbabilistic forecasts\n", "suffix": ".\nThe measure is calculated for ", "start": 17832, "end": 17939, "artefact": "A9"}]; kind summary] The M5 uncertainty competition scored quantile forecasts with the scaled pinball loss: each series' [[D12:pinball loss]] divided by the in-sample mean absolute error of its one-step naive forecast. scope: data_type: quantile forecasts in the M5 uncertainty competition C49 [methodological; supporting; synthesis; support: cited B18; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: kind summary] The M4 competition scored prediction intervals with the mean scaled interval score: the [[D13:interval score]] of each series scaled by an in-sample error of that series. scope: data_type: prediction intervals in the M4 competition C50 [methodological; supporting; synthesis; support: cited B5, at Section 3.1 (Comparisons of two classifiers); limitations L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B5", "locator": "Section 3.1 (Comparisons of two classifiers)", "lab_event": "XT23", "exact": "Averages are also susceptible to outliers", "prefix": "ms\nwith similar properties.\n ", "suffix": ". They allow classifier’s excell", "start": 18817, "end": 18858, "artefact": "A4"}]; kind summary] Averages of a performance measure over many data sets are susceptible to outliers. scope: data_type: comparisons of methods over many data sets C51 [methodological; supporting; synthesis; support: cited B5, at Section 3.2.2 (Friedman test), cited B5, at Section 5 (Conclusion); limitations L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B5", "locator": "Section 3.2.2 (Friedman test)", "lab_event": "XT24", "exact": "It ranks the algorithms for each data set separately", "prefix": "of the repeated-measures\nANOVA. ", "suffix": ", the best performing algorithm ", "start": 38209, "end": 38261, "artefact": "A4"}, {"reference": "B5", "locator": "Section 3.2.2 (Friedman test)", "lab_event": "XT25", "exact": "The Nemenyi test", "prefix": "n proceed with a post-hoc test. ", "suffix": " (Ne-\nmenyi, 1963) is similar to", "start": 40625, "end": 40641, "artefact": "A4"}, {"reference": "B5", "locator": "Section 5 (Conclusion)", "lab_event": "XT26", "exact": "They are appropriate since they assume some, but limited commensurability", "prefix": " are suitable for our\nproblems. ", "suffix": ". They\nare safer than parametric", "start": 432529, "end": 432602, "artefact": "A4"}]; kind summary] The [[D25:Friedman test]], followed by the [[D26:Nemenyi test]], compares several methods over many data sets by their ranks on each data set, assuming only limited commensurability of the measure across data sets. scope: data_type: comparisons of several methods over many data sets C52 [methodological; supporting; synthesis; support: cited B15; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: kind summary] [[D27:MCB]] on ranks was used to test the differences between the methods of the M3 competition. scope: data_type: comparisons of the methods of the M3 competition C53 [methodological; supporting; synthesis; support: cited B16; limitations L5, L6; verifiability open; planning exploratory; outcome supported] [review: kind summary] The [[D28:Diebold-Mariano test]] of equal accuracy of two forecasts allows forecast errors that are serially correlated and not normally distributed. scope: data_type: two forecasts of one series WARRANTS W1 [deductive; premises C30] A forecast is built to target one functional, and only a scoring function consistent for that functional is lowest in expectation for it; a scoring function consistent for another functional ranks forecasts by how close they come to that other functional. W2 [deductive; premises C29, C30] Squared error is consistent for the mean, so its expected value is lowest for the forecast that equals the mean; the RMSE, the square root of the mean squared error, ranks forecasts in the same order. W3 [deductive; premises C47, C29, C46] Dividing a series' squared errors by a fixed positive number, such as the in-sample mean squared error of its one-step naive forecast, does not change which forecast minimises their expected value, so the scaled RMSE (the RMSSE) keeps the target of the RMSE, the mean, and the scaled values of different series can be averaged, as in the M5 accuracy competition. W4 [deductive; premises C31, C29] Absolute error, which MAE averages and MASE scales, is consistent for the median, and squared error for the mean. When the mean and median forecasts coincide (a symmetric predictive distribution), both reward the same forecast; otherwise MAE and MASE reward the median forecast over a forecast that is correct for the mean. W5 [deductive; premises C32] MAPE is consistent for the median of order -1, which supports severe underforecasts compared with the mean or the median; it therefore rewards forecasts that are too low for either target. W6 [deductive; premises C33] sMAPE penalises forecasts below the outcome more heavily than forecasts above it, so it rewards forecasts that are too high; and it divides by the mean of the absolute outcome and the absolute forecast, which is zero when both are zero, so it is then undefined. W7 [deductive; premises C31, C46] Absolute error is consistent for the median, so its expected value is lowest for the median forecast; MASE divides each series' absolute errors by a fixed in-sample quantity, which keeps that target and lets series on different scales be averaged. W8 [deductive; premises C41, C31] On intermittent series the median of the outcome can be zero, and absolute error, consistent for the median, then gives its best score to a forecast of zero in every period, which says nothing of the demand to be met; a measure of the mean or of the whole distribution does not reward it. W9 [deductive; premises C34] The pinball loss at a level is consistent for the quantile at that level, so its expected value is lowest for the correct quantile; averaging over the levels gives each quantile the same weight. W10 [deductive; premises C35] The interval score is proper for central intervals and addresses width as well as coverage; coverage against the nominal level shows how far the intervals' calibration departs from what they claim. W11 [deductive; premises C35] Coverage counts only whether each outcome falls inside its interval; widening an interval never lowers its coverage, so coverage alone cannot penalise needless width, which the interval score does. W12 [deductive; premises C36, C37] The CRPS and the log score are proper: their expected values are best for the true distribution, so they never reward a forecaster for reporting a distribution other than the one believed. W13 [deductive; premises C39, C35] Predictive distributions are assessed by maximising their sharpness subject to their calibration: a sharper forecast that is not calibrated is overconfident, so sharpness tells which of two forecasts is better only when both are calibrated. A prediction interval summarises a predictive distribution, and the interval score weighs its width against its misses in the same way. W14 [deductive; premises C40, C36, C37] The probability integral transform of a discrete distribution takes few values and is not uniform even when the forecast is calibrated, so the calibration of count forecasts is checked with the randomised transform; proper scoring rules apply to count distributions as to any other. W15 [deductive; premises C38, C37] The Brier score is proper and the log score strictly proper, so both are best in expectation for the true probabilities of the classes. W16 [deductive; premises C38, C37] Accuracy uses a forecast probability only through which side of a threshold it falls, and counts every misclassification as the same loss; two forecasts with different probabilities but the same thresholded labels score the same, and unequal costs of errors are not reflected. The Brier and log scores, which are proper, reward the probabilities themselves. W17 [deductive; premises C42] The AUC averages the misclassification loss over a cost distribution that depends on the classifier, so different classifiers are compared with different weightings of the costs of errors; and it depends only on how the scores rank cases, not on whether the probabilities are right. W18 [deductive; premises C43] On imbalanced data the ROC curve can be deceptive about how reliable a classifier is, and the precision-recall curve is the more informative of the two; reporting it shows what the ROC curve can hide. W19 [deductive; premises C30] Each scoring function is lowest in expectation for the functional it is consistent for, so one measure favours the methods that target that functional; comparing methods that target different functionals with one measure compares their targets, not their skill. W20 [deductive; premises C46, C50] A scaled measure can be averaged over series, but averages over many data sets are susceptible to outliers; the median and the quartiles show how accuracy is spread over the series, which the mean alone hides. W21 [deductive; premises C51] The Friedman test, followed by the Nemenyi test, compares methods by their ranks on each data set and assumes only limited commensurability of the measure across data sets; it tells whether differences between methods whose aggregate accuracy is close are larger than chance. W22 [deductive; premises C52] Multiple comparisons with the best on ranks test which methods differ from the best one, and were used for this on the methods of the M3 competition. W23 [deductive; premises C53] The Diebold-Mariano test allows forecast errors that are serially correlated and not normally distributed, as the errors of multi-step forecasts of one series usually are; tests that assume independent, normal errors do not hold for them. W24 [deductive; premises C30] A measure fits a study when it is consistent for what the compared methods predict; that a competition or a well-known study used another measure says nothing about whether that one is consistent for the forecasts of this study. W25 [deductive; premises C44, C45] Scale-dependent measures cannot be compared across series on different scales, and summed or averaged without scaling they are dominated by the series with the largest values; the result then says little about accuracy on the other series. W26 [deductive; premises C44, C46, C47] Scaled errors are independent of the scale of the data and can be averaged over series; dividing by a quantity of the series that does not depend on the compared forecasts keeps the target that the unscaled measure is consistent for. W27 [deductive; premises C48, C34, C44] Pinball losses of series on different scales cannot be compared unscaled; dividing each series' pinball loss by a quantity of that series that does not depend on the forecasts keeps the quantile it is consistent for, as the scaled pinball loss of the M5 uncertainty competition does. W28 [deductive; premises C49, C35, C44] Interval scores of series on different scales cannot be compared unscaled; dividing each series' interval score by a positive quantity of that series that does not depend on the forecasts keeps it proper, as the mean scaled interval score of the M4 competition does. LIMITATIONS L1 [data_quality; affects C42, C17; could_reverse; mitigation none] These claims rest only on works checked by their abstracts and bibliographic records (Brier 1950, Hand 2009: lab notes NT7 and NT8); a full text could qualify what the abstract states. L2 [data_quality; affects C38; reduces_precision; mitigation none] Part of the support of these claims comes from a work checked only by its abstract (Brier 1950, lab note NT7); the rest comes from works read in full. L3 [measurement; affects C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C44, C46; reduces_precision; mitigation none] Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions. L4 [other; affects C16; could_reverse; mitigation none] No cited work states this recommendation or its reason; it rests on an argument from the definition of accuracy (its warrant), with the properness of the Brier and log scores as premises. L5 [researcher_position; affects C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C40, C41, C42, C43, C44, C45, C46, C47, C48, C49, C50, C51, C52, C53; could_bias_magnitude; mitigation none] The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. L6 [data_quality; affects C30, C33, C40, C41, C45, C47, C48, C49, C52, C53; reduces_precision; mitigation none] These claims rest in part on works read in full by the guarantor, not by the agent that wrote the claimset (lab note NT14); no passage of those works is quoted, so a reader checks these claims against the works themselves. ARTEFACTS A1 [data] Making and evaluating point forecasts (arXiv:0912.0902, the preprint of Gneiting 2011, JASA 106(494), 746-762, doi:10.1198/jasa.2011.r10138) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR1 on 2026-11-13. venue: arXiv. Written by others. A2 [data] Strictly proper scoring rules, prediction, and estimation (Gneiting and Raftery 2007, JASA 102(477), 359-378, doi:10.1198/016214506000001437), author's copy (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR2 on 2026-11-13. venue: Journal of the American Statistical Association. Written by others. A3 [data] Probabilistic forecasts, calibration and sharpness (Gneiting, Balabdaoui and Raftery 2007, JRSS B 69(2), 243-268, doi:10.1111/j.1467-9868.2007.00587.x), author's copy (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR3 on 2026-11-13. venue: Journal of the Royal Statistical Society B. Written by others. A4 [data] Statistical comparisons of classifiers over multiple data sets (Demsar 2006, JMLR 7, 1-30) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR4 on 2026-11-13. venue: Journal of Machine Learning Research. Written by others. A7 [data] Another look at measures of forecast accuracy (preprint of Hyndman and Koehler 2006, IJF 22(4), 679-688, doi:10.1016/j.ijforecast.2006.03.001) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR7 on 2026-11-13. venue: International Journal of Forecasting. Written by others. A8 [data] The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets (Saito and Rehmsmeier 2015, PLoS ONE 10(3), e0118432, doi:10.1371/journal.pone.0118432) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR8 on 2026-11-13. venue: PLoS ONE; published: 2015-03-04. Written by others. A9 [data] The M5 Competition: Competitors' Guide (the organisers' definition of the RMSSE, the scaled pinball loss and the reasons for them) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR9 on 2026-11-13. venue: M Open Forecasting Center, GitHub repository Mcompetitions/M5-methods. Written by others. A10 [lab_record] Lab record of the study: questions, hypotheses, plans, runs, results and decisions, as an append-only, hash-chained event log (Claimset lab record specification v0.22, begun 2026-11-13; exported with claimset 0.43.0). REFERENCES B1 https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf (url) B2 10.1198/jasa.2011.r10138 (doi) B3 10.1198/016214506000001437 (doi) B4 10.1111/j.1467-9868.2007.00587.x (doi) B5 https://jmlr.org/papers/v7/demsar06a.html (url) B6 10.1016/j.ijforecast.2006.03.001 (doi) B7 10.1371/journal.pone.0118432 (doi) B8 10.1016/j.ijforecast.2019.02.017 (doi) B9 10.1016/j.ijforecast.2021.11.013 (doi) B10 10.1016/j.ijforecast.2021.10.009 (doi) B11 10.1016/S0169-2070(99)00007-2 (doi) B12 10.1016/j.ijforecast.2015.12.004 (doi) B13 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 (doi) B14 10.1007/s10994-009-5119-5 (doi) B15 10.1016/j.ijforecast.2004.10.003 (doi) B16 10.1080/07350015.1995.10524599 (doi) B17 10.1016/j.ijforecast.2022.08.003 (doi) B18 10.1016/j.ijforecast.2019.04.014 (doi) SCORES Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. Support of the paper: 78 (medium), the mean of its 28 principal claims; the lowest is C17 at 64. C1: Support 79 (medium); novelty: summarises earlier work, by the authors C2: Support 79 (medium); novelty: summarises earlier work, by the authors C3: Support 79 (medium); novelty: summarises earlier work, by the authors C4: Support 80 (high); novelty: summarises earlier work, by the authors C5: Support 80 (high); novelty: summarises earlier work, by the authors C6: Support 79 (medium); novelty: summarises earlier work, by the authors C7: Support 80 (high); novelty: summarises earlier work, by the authors C8: Support 79 (medium); novelty: summarises earlier work, by the authors C9: Support 80 (high); novelty: summarises earlier work, by the authors C10: Support 80 (high); novelty: summarises earlier work, by the authors C11: Support 80 (high); novelty: summarises earlier work, by the authors · in question: close match on EntryLib not acknowledged C12: Support 80 (high); novelty: summarises earlier work, by the authors C13: Support 80 (high); novelty: summarises earlier work, by the authors C14: Support 74 (medium); novelty: summarises earlier work, by the authors C15: Support 79 (medium); novelty: summarises earlier work, by the authors C16: Support 72 (medium); novelty: extends earlier work, by the authors C17: Support 64 (medium); novelty: summarises earlier work, by the authors C18: Support 81 (high); novelty: summarises earlier work, by the authors C19: Support 79 (medium); novelty: summarises earlier work, by the authors C20: Support 80 (high); novelty: summarises earlier work, by the authors C21: Support 81 (high); novelty: summarises earlier work, by the authors C22: Support 75 (medium); novelty: summarises earlier work, by the authors C23: Support 75 (medium); novelty: summarises earlier work, by the authors C24: Support 79 (medium); novelty: summarises earlier work, by the authors C25: Support 74 (medium); novelty: summarises earlier work, by the authors C26: Support 79 (medium); novelty: summarises earlier work, by the authors C27: Support 79 (medium); novelty: summarises earlier work, by the authors C28: Support 74 (medium); novelty: summarises earlier work, by the authors C29: Support 93 (high); novelty: summarises earlier work, by the authors C30: Support 92 (high); novelty: summarises earlier work, by the authors C31: Support 93 (high); novelty: summarises earlier work, by the authors C32: Support 93 (high); novelty: summarises earlier work, by the authors C33: Support 92 (high); novelty: summarises earlier work, by the authors C34: Support 93 (high); novelty: summarises earlier work, by the authors C35: Support 93 (high); novelty: summarises earlier work, by the authors C36: Support 93 (high); novelty: summarises earlier work, by the authors C37: Support 93 (high); novelty: summarises earlier work, by the authors C38: Support 92 (high); novelty: summarises earlier work, by the authors C39: Support 93 (high); novelty: summarises earlier work, by the authors C40: Support 78 (medium); novelty: summarises earlier work, by the authors C41: Support 93 (high); novelty: summarises earlier work, by the authors C42: Support 67 (medium); novelty: summarises earlier work, by the authors C43: Support 94 (high); novelty: summarises earlier work, by the authors C44: Support 93 (high); novelty: summarises earlier work, by the authors C45: Support 78 (medium); novelty: summarises earlier work, by the authors C46: Support 93 (high); novelty: summarises earlier work, by the authors C47: Support 93 (high); novelty: summarises earlier work, by the authors C48: Support 93 (high); novelty: summarises earlier work, by the authors C49: Support 78 (medium); novelty: summarises earlier work, by the authors C50: Support 94 (high); novelty: summarises earlier work, by the authors C51: Support 94 (high); novelty: summarises earlier work, by the authors C52: Support 78 (medium); novelty: summarises earlier work, by the authors C53: Support 78 (medium); novelty: summarises earlier work, by the authors Open problems in the principal claims and what they rest on: P1 [support; up to -20; C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28, C40, C42, C45, C49, C52, C53] Rests on cited work only; cited work read in full, without the passage it rests on; an argument from other claims. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review). P2 [open; up to -10; C16] Limitation L4 (could reverse, not mitigated): No cited work states this recommendation or its reason; it rests on an argument from the definition of accuracy (its warrant), with the properness of the Brier and log scores as premises. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies. P3 [open; up to -10; C17, C42] Limitation L1 (could reverse, not mitigated): These claims rest only on works checked by their abstracts and bibliographic records (Brier 1950, Hand 2009: lab notes NT7 and NT8); a full text could qualify what the abstract states. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies. P4 [plan; up to -3; C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28] The review's search for the literature: none is recorded; the claims were confirmed against works the authors knew. To remove it: Record the protocol, then the searches and the screening of what they found (claimset lab protocol, search, screen), and export them. P5 [open; up to -1; C15, C16, C38] Limitation L2 (reduces precision, not mitigated): Part of the support of these claims comes from a work checked only by its abstract (Brier 1950, lab note NT7); the rest comes from works read in full. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies. P6 [open; up to -1; C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C19, C20, C24, C25, C26, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C44, C46] Limitation L3 (reduces precision, not mitigated): Section and theorem numbers for Gneiting 2011, Gneiting and Raftery 2007, Gneiting, Balabdaoui and Raftery 2007 and Hyndman and Koehler 2006 were read in their preprints or authors' copies (arXiv:0912.0902 and the authors' web pages), whose numbering may differ from the published versions. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies. P7 [open; up to -6; C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C40, C41, C42, C43, C44, C45, C46, C47, C48, C49, C50, C51, C52, C53] Limitation L5 (could bias magnitude, not mitigated): The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies. P8 [open; up to -1; C1, C2, C3, C6, C8, C14, C19, C22, C23, C24, C25, C26, C27, C28, C30, C33, C40, C41, C45, C47, C48, C49, C52, C53] Limitation L6 (reduces precision, not mitigated): These claims rest in part on works read in full by the guarantor, not by the agent that wrote the claimset (lab note NT14); no passage of those works is quoted, so a reader checks these claims against the works themselves. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies. HOW TO CITE This claimset: Svetunkov, I. (guarantor) (2026). Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2. https://entrylib.org/p/2026.000008@v2 One claim of it: by its identifier, as entrylib:2026.000008@v2#C1 for claim C1. Each principal claim has its own page, as https://entrylib.org/p/2026.000008/C1.