LAB RECORD: Evaluating forecasts and classifiers: which measures, scoring rules and tests to use Study: prediction-good-practice · Lab record specification v0.22 · Begun 2026-11-13 · Repository: https://github.com/entrylib/study-planning Events: 298; runs: 0; results: 0 (kept in files, never listed here); head: seq 298, sha256:17bfd0887168097d0159bcaa8ba9a209ed91afb60c38fbe597fbb80370a9de47 This view is made from the log; the log is the record (lab record §11). WHO - AC1 Claude Code (agent; Anthropic Claude Opus 5.5; in Claude Code; literature summary), main log; credit: conceptualization, methodology, data_curation - AC2 Ivan Svetunkov (human; ORCID 0000-0001-7826-0281), main log QUESTIONS AND HYPOTHESES Q1 Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… H1 [supported] Point forecasts should be evaluated with a [[TM1:consistent scoring function]] for the functional they target, such as the mean, the median or a quantile. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM1 term by AC1: consistent scoring function [concept] 2026-11-13 13:20 H1 hypothesis by AC1: To evaluate point forecasts, use a [[TM1:consistent scoring function]] for the functional the forecasts target (the mean, the median or a quantile): a scoring… [a_priori] 2026-11-13 13:30 ST1 status by AC1: H1 -> supported, evidence SR1, NT2: Gneiting (2011, abstract, SR1): 'If the forecaster receives a directive in the form of a functional, it is critical tha… 2026-11-13 16:16 H37 hypothesis by AC1: Which point forecast has the lowest expected score depends on the scoring function: a [[TM1:consistent scoring function]] for a functional, such as the mean or… [a_priori] 2026-11-13 16:20 H62 hypothesis by AC1: Point forecasts should be evaluated with a [[TM1:consistent scoring function]] for the functional they target: the mean, the median or a quantile. [a_priori] 2026-11-13 16:20 ST69 status by AC1: H62 -> supported, evidence H37: A forecast is built to target one functional, and only a scoring function consistent for that functional is lowest in e… 2026-11-13 16:25 H90 hypothesis by AC1: Point forecasts should be evaluated with a [[TM1:consistent scoring function]] for the functional they target, such as the mean, the median or a quantile. [a_priori] 2026-11-13 16:25 ST97 status by AC1: H90 -> supported, evidence H37: A forecast is built to target one functional, and only a scoring function consistent for that functional is lowest in e… H2 [supported] [[TM3:Point forecasts of the mean]] should be evaluated with squared error, as in [[TM5:RMSE]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM3 term by AC1: point forecast of the mean [concept] 2026-11-13 13:18 TM5 term by AC1: RMSE [variable] 2026-11-13 13:20 H2 hypothesis by AC1: Point forecasts of the mean ([[TM3:point forecast of the mean]]) should be evaluated with squared error ([[TM5:RMSE]]), the scoring function consistent for the… [a_priori] 2026-11-13 13:30 ST2 status by AC1: H2 -> supported, evidence SR1: Gneiting (2011, SR1): 'the squared error scoring function is consistent for the mean' (Section 5, Discussion); the scor… 2026-11-13 16:16 H36 hypothesis by AC1: Squared error is a [[TM1:consistent scoring function]] for the mean. [a_priori] 2026-11-13 16:16 H37 hypothesis by AC1: Which point forecast has the lowest expected score depends on the scoring function: a [[TM1:consistent scoring function]] for a functional, such as the mean or… [a_priori] 2026-11-13 16:20 H63 hypothesis by AC1: [[TM3:Point forecasts of the mean]] should be evaluated with squared error, as in [[TM5:RMSE]]. [a_priori] 2026-11-13 16:20 ST70 status by AC1: H63 -> supported, evidence H36, H37: Squared error is consistent for the mean, so its expected value is lowest for the forecast that equals the mean; the RM… H3 [supported] When [[TM3:point forecasts of the mean]] are compared over series on different scales, each series' [[TM5:RMSE]] should be scaled by the in-sample RMSE of its one-step naive forecast, as in [[TM6:RMSSE]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM3 term by AC1: point forecast of the mean [concept] 2026-11-13 13:18 TM5 term by AC1: RMSE [variable] 2026-11-13 13:18 TM6 term by AC1: RMSSE [variable] 2026-11-13 13:20 H3 hypothesis by AC1: When [[TM3:point forecasts of the mean]] are compared over series on different scales, the [[TM5:RMSE]] of each series should be scaled by the in-sample RMSE o… [a_priori] 2026-11-13 13:30 ST3 status by AC1: H3 -> supported, evidence NT3, SR1: The M5 accuracy competition (NT3) ranked point forecasts by the RMSSE: the RMSE of each series scaled by the in-sample… 2026-11-13 14:08 ST34 status by AC1: H3 -> supported, evidence SR9, NT3, SR1: The M5 Competitors' Guide (SR9, Point forecasts): the RMSSE divides each series' squared errors by the in-sample mean s… 2026-11-13 16:16 H36 hypothesis by AC1: Squared error is a [[TM1:consistent scoring function]] for the mean. [a_priori] 2026-11-13 16:18 H53 hypothesis by AC1: Scaled errors, as in [[TM8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. [a_priori] 2026-11-13 16:18 H54 hypothesis by AC1: The M5 accuracy competition scored point forecasts with the [[TM6:RMSSE]]: each series' squared errors divided by the in-sample mean squared error of its one-s… [a_priori] 2026-11-13 16:20 H64 hypothesis by AC1: When [[TM3:point forecasts of the mean]] are compared over series on different scales, each series' [[TM5:RMSE]] should be scaled by the in-sample RMSE of its… [a_priori] 2026-11-13 16:20 ST71 status by AC1: H64 -> supported, evidence H54, H36, H53: Dividing a series' squared errors by a fixed positive number, the in-sample mean squared error of its one-step naive fo… H4 [abandoned] The point forecasts of [[TM30:ETS]] models are [[TM3:point forecasts of the mean]] (conditional means of the forecast distribution), except, beyond one step ahead, for models with a multiplicative trend or multiplicative seasonality. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM3 term by AC1: point forecast of the mean [concept] 2026-11-13 13:19 TM30 term by AC1: ETS [method] 2026-11-13 13:20 H4 hypothesis by AC1: The point forecasts of [[TM30:ETS]] models are [[TM3:point forecasts of the mean]] (conditional means of the forecast distribution), except, beyond one step ah… [a_priori] 2026-11-13 13:30 ST4 status by AC1: H4 -> supported, evidence SR5: Hyndman and Khandakar (2008, SR5, Section 2.2): for many models the conditional means are identical to the point foreca… 2026-11-13 13:44 DC1 decision by AC1: The supervisor left the implementation facts out of the study: 'Implementation facts are not needed here. This is a side thing, which belongs to ETS, not this… 2026-11-13 13:44 ST28 status by AC1: H4 -> abandoned: Out of the study's scope by the supervisor's decision DC1: a property of particular models, not a rule for evaluating p… H5 [abandoned] Point forecasts made on a Box-Cox or log scale and transformed back to the original scale are [[TM4:point forecasts of the median]], not of the mean, unless they are bias-adjusted ([[TM29:bias adjustment]]). 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM4 term by AC1: point forecast of the median [concept] 2026-11-13 13:19 TM29 term by AC1: bias adjustment [method] 2026-11-13 13:20 H5 hypothesis by AC1: Point forecasts made on a Box-Cox or log scale and transformed back to the original scale are [[TM4:point forecasts of the median]], not of the mean, unless th… [a_priori] 2026-11-13 13:30 ST5 status by AC1: H5 -> supported, evidence SR6: Hyndman and Athanasopoulos (2018, SR6, Section 3.2): 'the back-transformed point forecast will not be the mean of the f… 2026-11-13 13:44 DC1 decision by AC1: The supervisor left the implementation facts out of the study: 'Implementation facts are not needed here. This is a side thing, which belongs to ETS, not this… 2026-11-13 13:44 ST29 status by AC1: H5 -> abandoned: Out of the study's scope by the supervisor's decision DC1: a property of particular models, not a rule for evaluating p… H6 [supported] [[TM7:MAE]] and [[TM8:MASE]] should not be the main measure for [[TM3:point forecasts of the mean]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM3 term by AC1: point forecast of the mean [concept] 2026-11-13 13:18 TM7 term by AC1: MAE [variable] 2026-11-13 13:18 TM8 term by AC1: MASE [variable] 2026-11-13 13:20 H6 hypothesis by AC1: [[TM7:MAE]] and [[TM8:MASE]] should not be the main measure for [[TM3:point forecasts of the mean]]: absolute error is consistent for the median and rewards me… [a_priori] 2026-11-13 13:30 ST6 status by AC1: H6 -> supported, evidence SR1: Gneiting (2011, SR1, Section 5): squared error is consistent for the mean, 'and the absolute error scoring function for… 2026-11-13 16:16 H36 hypothesis by AC1: Squared error is a [[TM1:consistent scoring function]] for the mean. [a_priori] 2026-11-13 16:16 H38 hypothesis by AC1: Absolute error is a [[TM1:consistent scoring function]] for the median. [a_priori] 2026-11-13 16:20 H65 hypothesis by AC1: [[TM7:MAE]] and [[TM8:MASE]] should not be the main measure for [[TM3:point forecasts of the mean]]. [a_priori] 2026-11-13 16:20 ST72 status by AC1: H65 -> supported, evidence H38, H36: Absolute error, which MAE averages and MASE scales, is consistent for the median, not the mean; on skewed outcomes the… H7 [supported] [[TM9:MAPE]] should not be the measure for point forecasts of the mean or the median. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM9 term by AC1: MAPE [variable] 2026-11-13 13:20 H7 hypothesis by AC1: [[TM9:MAPE]] should not be the measure for point forecasts of the mean or the median: it is consistent for the [[TM11:median of order -1]], which lies below th… [a_priori] 2026-11-13 13:30 ST7 status by AC1: H7 -> supported, evidence SR1: Gneiting (2011, SR1, Section 5): the absolute percentage error 'is consistent for a non-standard functional, namely, th… 2026-11-13 13:32 ST27 status by AC1: H7 -> supported, evidence SR1: Gneiting (2011, SR1, Section 5): the absolute percentage error 'is consistent for a non-standard functional, namely, th… 2026-11-13 16:16 H39 hypothesis by AC1: The absolute percentage error averaged by [[TM9:MAPE]] is consistent for the [[TM11:median of order -1]], a functional that supports severe underforecasts comp… [a_priori] 2026-11-13 16:20 H66 hypothesis by AC1: [[TM9:MAPE]] should not be the measure for point forecasts of the mean or the median. [a_priori] 2026-11-13 16:20 ST73 status by AC1: H66 -> supported, evidence H39: MAPE is consistent for the median of order -1, which supports severe underforecasts compared with the mean or the media… H8 [supported] [[TM10:sMAPE]] should not be the measure for point forecasts. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM10 term by AC1: sMAPE [variable] 2026-11-13 13:20 H8 hypothesis by AC1: [[TM10:sMAPE]] should not be the measure for point forecasts: it penalises errors of the same size differently depending on their sign, and it is undefined whe… [a_priori] 2026-11-13 13:30 ST8 status by AC1: H8 -> supported, evidence SR7, NT5: Hyndman and Koehler (2006, SR7, Section 2): the 'symmetric' measures 'are not as symmetric as their name suggests': for… 2026-11-13 13:45 H27 hypothesis by AC1: [[TM10:sMAPE]] should not be the measure for point forecasts: for the same outcome it penalises forecasts below the outcome more heavily than forecasts above i… [a_priori] 2026-11-13 13:51 ST30 status by AC1: H27 -> supported, evidence SR7, NT5: Hyndman and Koehler (2006, SR7, Section 2.2): for the same outcome, sMAPE 'has a heavier penalty when forecasts are low… 2026-11-13 16:17 H40 hypothesis by AC1: For the same outcome, [[TM10:sMAPE]] penalises a forecast below the outcome more heavily than a forecast above it. [a_priori] 2026-11-13 16:20 H61 hypothesis by AC1: [[TM10:sMAPE]] should not be the measure for point forecasts. [a_priori] 2026-11-13 16:20 ST68 status by AC1: H61 -> supported, evidence H40: sMAPE penalises forecasts below the outcome more heavily than forecasts above it, so it rewards forecasts that are too… H9 [supported] [[TM4:Point forecasts of the median]] should be evaluated with absolute error, as in [[TM7:MAE]] or, across series on different scales, [[TM8:MASE]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM4 term by AC1: point forecast of the median [concept] 2026-11-13 13:18 TM7 term by AC1: MAE [variable] 2026-11-13 13:18 TM8 term by AC1: MASE [variable] 2026-11-13 13:20 H9 hypothesis by AC1: Point forecasts of the median ([[TM4:point forecast of the median]]) should be evaluated with absolute error ([[TM7:MAE]]), the scoring function consistent for… [a_priori] 2026-11-13 13:30 ST9 status by AC1: H9 -> supported, evidence SR1, SR7: Gneiting (2011, SR1, Section 5): 'the absolute error scoring function' is consistent 'for the median'. Hyndman and Koeh… 2026-11-13 16:16 H38 hypothesis by AC1: Absolute error is a [[TM1:consistent scoring function]] for the median. [a_priori] 2026-11-13 16:18 H53 hypothesis by AC1: Scaled errors, as in [[TM8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. [a_priori] 2026-11-13 16:20 H67 hypothesis by AC1: [[TM4:Point forecasts of the median]] should be evaluated with absolute error, as in [[TM7:MAE]] or, across series on different scales, [[TM8:MASE]]. [a_priori] 2026-11-13 16:20 ST74 status by AC1: H67 -> supported, evidence H38, H53: Absolute error is consistent for the median, so its expected value is lowest for the median forecast; MASE divides each… H10 [supported] On [[TM24:intermittent series]], absolute-error measures should not be used alone. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM24 term by AC1: intermittent series [data] 2026-11-13 13:20 H10 hypothesis by AC1: On [[TM24:intermittent series]], absolute-error measures should not be used alone: the median of the outcome can be zero, and absolute error then rewards a for… [a_priori] 2026-11-13 13:30 ST10 status by AC1: H10 -> supported, evidence NT6, NT3: Kolassa (2016, NT6): on intermittent count data, absolute-error measures are minimised by a forecast of zero and are un… 2026-11-13 14:08 ST35 status by AC1: H10 -> supported, evidence SR9, NT6: The M5 Competitors' Guide (SR9, Point forecasts): on intermittent series 'absolute errors, which are optimized for the… 2026-11-13 16:16 H38 hypothesis by AC1: Absolute error is a [[TM1:consistent scoring function]] for the median. [a_priori] 2026-11-13 16:18 H48 hypothesis by AC1: On [[TM24:intermittent series]], absolute-error measures give their best scores to forecasts close to zero. [a_priori] 2026-11-13 16:20 H68 hypothesis by AC1: On [[TM24:intermittent series]], absolute-error measures should not be used alone. [a_priori] 2026-11-13 16:21 ST75 status by AC1: H68 -> supported, evidence H48, H38: On intermittent series the median of the outcome can be zero, and absolute error, consistent for the median, then gives… H11 [supported] Quantile forecasts should be evaluated with the [[TM12:pinball loss]] at the quantile's level, averaged over the levels when there are several. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM12 term by AC1: pinball loss [variable] 2026-11-13 13:20 H11 hypothesis by AC1: Quantile forecasts should be evaluated with the [[TM12:pinball loss]] at the quantile's level, the scoring function consistent for that quantile, averaged over… [a_priori] 2026-11-13 13:30 ST11 status by AC1: H11 -> supported, evidence SR1, NT4: Gneiting (2011, SR1, abstract and Theorem 3.3): a scoring function 'is consistent for a quantile if and only if it is g… 2026-11-13 14:07 H30 hypothesis by AC1: Quantile forecasts should be evaluated with the [[TM12:pinball loss]] at the quantile's level, the scoring function consistent for that quantile, averaged over… [a_priori] 2026-11-13 14:08 ST36 status by AC1: H30 -> supported, evidence SR1: Gneiting (2011, SR1, abstract and Theorem 3.3): a scoring function 'is consistent for a quantile if and only if it is g… 2026-11-13 16:17 H41 hypothesis by AC1: The [[TM12:pinball loss]] at a level is a [[TM1:consistent scoring function]] for the quantile at that level. [a_priori] 2026-11-13 16:21 H83 hypothesis by AC1: Quantile forecasts should be evaluated with the [[TM12:pinball loss]] at the quantile's level, averaged over the levels when there are several. [a_priori] 2026-11-13 16:21 ST90 status by AC1: H83 -> supported, evidence H41: The pinball loss at a level is consistent for the quantile at that level, so its expected value is lowest for the corre… H12 [supported] Prediction intervals should be evaluated with the [[TM13:interval score]], reported together with their [[TM14:coverage]] against the nominal level. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM13 term by AC1: interval score [variable] 2026-11-13 13:19 TM14 term by AC1: coverage [variable] 2026-11-13 13:20 H12 hypothesis by AC1: Prediction intervals should be evaluated with the [[TM13:interval score]], a proper scoring rule for central intervals, reported together with their [[TM14:cov… [a_priori] 2026-11-13 13:31 ST12 status by AC1: H12 -> supported, evidence SR2: Gneiting and Raftery (2007, SR2, Section 1 and Section 6.2): they 'discuss the interval score, a scoring rule for predi… 2026-11-13 16:17 H42 hypothesis by AC1: The [[TM13:interval score]] is a [[TM2:proper scoring rule]] for central prediction intervals that addresses their width as well as their coverage. [a_priori] 2026-11-13 16:21 H69 hypothesis by AC1: Prediction intervals should be evaluated with the [[TM13:interval score]], reported together with their [[TM14:coverage]] against the nominal level. [a_priori] 2026-11-13 16:21 ST76 status by AC1: H69 -> supported, evidence H42: The interval score is proper for central intervals and addresses width as well as coverage; coverage against the nomina… H13 [supported] [[TM14:Coverage]] alone should not be the measure of prediction intervals. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM14 term by AC1: coverage [variable] 2026-11-13 13:20 H13 hypothesis by AC1: [[TM14:coverage]] alone should not be the measure of prediction intervals: it does not measure their width, so sharp intervals and needlessly wide ones can rea… [a_priori] 2026-11-13 13:31 ST13 status by AC1: H13 -> supported, evidence SR2: Gneiting and Raftery (2007, SR2, abstract) propose the interval score because it 'addresses width as well as coverage':… 2026-11-13 16:17 H42 hypothesis by AC1: The [[TM13:interval score]] is a [[TM2:proper scoring rule]] for central prediction intervals that addresses their width as well as their coverage. [a_priori] 2026-11-13 16:21 H70 hypothesis by AC1: [[TM14:Coverage]] alone should not be the measure of prediction intervals. [a_priori] 2026-11-13 16:21 ST77 status by AC1: H70 -> supported, evidence H42: Coverage counts only whether each outcome falls inside its interval; widening an interval never lowers its coverage, so… H14 [supported] Predictive distributions should be evaluated with a [[TM2:proper scoring rule]], such as the [[TM15:CRPS]] or the [[TM16:log score]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM2 term by AC1: proper scoring rule [concept] 2026-11-13 13:19 TM15 term by AC1: CRPS [variable] 2026-11-13 13:19 TM16 term by AC1: log score [variable] 2026-11-13 13:20 H14 hypothesis by AC1: Predictive distributions should be evaluated with a [[TM2:proper scoring rule]], such as the [[TM15:CRPS]] or the [[TM16:log score]]. [a_priori] 2026-11-13 13:31 ST14 status by AC1: H14 -> supported, evidence SR2: Gneiting and Raftery (2007, SR2): 'The CRPS is proper relative to the class P' (Section 4.2) and 'The logarithmic score… 2026-11-13 16:17 H43 hypothesis by AC1: The [[TM15:CRPS]] is a [[TM2:proper scoring rule]] for predictive distributions of continuous outcomes. [a_priori] 2026-11-13 16:17 H44 hypothesis by AC1: The [[TM16:log score]] is a strictly [[TM2:proper scoring rule]]. [a_priori] 2026-11-13 16:21 H71 hypothesis by AC1: Predictive distributions should be evaluated with a [[TM2:proper scoring rule]], such as the [[TM15:CRPS]] or the [[TM16:log score]]. [a_priori] 2026-11-13 16:21 ST78 status by AC1: H71 -> supported, evidence H43, H44: The CRPS and the log score are proper: their expected values are best for the true distribution, so they never reward a… H15 [supported] The evaluation of predictive distributions should check their [[TM19:calibration]] and report their [[TM20:sharpness]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM19 term by AC1: calibration [concept] 2026-11-13 13:19 TM20 term by AC1: sharpness [concept] 2026-11-13 13:20 H15 hypothesis by AC1: The evaluation of predictive distributions should check their [[TM19:calibration]], with the [[TM18:probability integral transform]] or coverage at several lev… [a_priori] 2026-11-13 13:31 ST15 status by AC1: H15 -> supported, evidence SR3: Gneiting, Balabdaoui and Raftery (2007, SR3, abstract): evaluation 'based on the paradigm of maximizing the sharpness o… 2026-11-13 16:17 H46 hypothesis by AC1: Predictive distributions are assessed by maximising their [[TM20:sharpness]] subject to their [[TM19:calibration]]. [a_priori] 2026-11-13 16:21 H72 hypothesis by AC1: The evaluation of predictive distributions should check their [[TM19:calibration]] and report their [[TM20:sharpness]]. [a_priori] 2026-11-13 16:21 ST79 status by AC1: H72 -> supported, evidence H46: Predictive distributions are assessed by maximising their sharpness subject to calibration; checking calibration, with… H16 [supported] Predictive distributions of counts, such as forecasts of intermittent sales, should be evaluated with proper scoring rules and a randomised [[TM18:probability integral transform]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM18 term by AC1: probability integral transform [concept] 2026-11-13 13:20 H16 hypothesis by AC1: Predictive distributions of counts, such as forecasts of intermittent sales, should be evaluated with proper scoring rules and a randomised [[TM18:probability… [a_priori] 2026-11-13 13:31 ST16 status by AC1: H16 -> supported, evidence NT6: Kolassa (2016, NT6): predictive distributions of count data are evaluated with proper scoring rules and a randomised pr… 2026-11-13 16:17 H43 hypothesis by AC1: The [[TM15:CRPS]] is a [[TM2:proper scoring rule]] for predictive distributions of continuous outcomes. [a_priori] 2026-11-13 16:17 H44 hypothesis by AC1: The [[TM16:log score]] is a strictly [[TM2:proper scoring rule]]. [a_priori] 2026-11-13 16:18 H47 hypothesis by AC1: The calibration of predictive distributions of counts can be assessed with a randomised [[TM18:probability integral transform]]. [a_priori] 2026-11-13 16:21 H73 hypothesis by AC1: Predictive distributions of counts, such as forecasts of intermittent sales, should be evaluated with proper scoring rules and a randomised [[TM18:probability… [a_priori] 2026-11-13 16:21 ST80 status by AC1: H73 -> supported, evidence H47, H43, H44: The probability integral transform of a discrete distribution takes few values and is not uniform even when the forecas… H17 [supported] Forecast probabilities of classes should be evaluated with a [[TM2:proper scoring rule]], such as the [[TM16:log score]] or the [[TM17:Brier score]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM2 term by AC1: proper scoring rule [concept] 2026-11-13 13:19 TM16 term by AC1: log score [variable] 2026-11-13 13:19 TM17 term by AC1: Brier score [variable] 2026-11-13 13:20 H17 hypothesis by AC1: Forecast probabilities of classes should be evaluated with a [[TM2:proper scoring rule]], such as the [[TM16:log score]] or the [[TM17:Brier score]]. [a_priori] 2026-11-13 13:31 ST17 status by AC1: H17 -> supported, evidence SR2, NT7: Gneiting and Raftery (2007, SR2, Section 3, Example 1, 'Quadratic or Brier score', and the logarithmic score): both are… 2026-11-13 16:17 H44 hypothesis by AC1: The [[TM16:log score]] is a strictly [[TM2:proper scoring rule]]. [a_priori] 2026-11-13 16:17 H45 hypothesis by AC1: The [[TM17:Brier score]] is a [[TM2:proper scoring rule]] for forecast probabilities of classes. [a_priori] 2026-11-13 16:21 H74 hypothesis by AC1: Forecast probabilities of classes should be evaluated with a [[TM2:proper scoring rule]], such as the [[TM16:log score]] or the [[TM17:Brier score]]. [a_priori] 2026-11-13 16:21 ST81 status by AC1: H74 -> supported, evidence H45, H44: The Brier score is proper and the log score strictly proper, so both are best in expectation for the true probabilities… H18 [supported] [[TM21:Accuracy]] should not be the measure of forecast class probabilities. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM21 term by AC1: accuracy [variable] 2026-11-13 13:20 H18 hypothesis by AC1: [[TM21:accuracy]] should not be the measure of forecast class probabilities: it ignores the probabilities beyond which side of a threshold they fall, and weigh… [a_priori] 2026-11-13 13:31 ST18 status by AC1: H18 -> supported, evidence NT11: The argument of NT11: accuracy uses a probability only through which side of a threshold it falls and counts every erro… 2026-11-13 16:17 H44 hypothesis by AC1: The [[TM16:log score]] is a strictly [[TM2:proper scoring rule]]. [a_priori] 2026-11-13 16:17 H45 hypothesis by AC1: The [[TM17:Brier score]] is a [[TM2:proper scoring rule]] for forecast probabilities of classes. [a_priori] 2026-11-13 16:21 H75 hypothesis by AC1: [[TM21:Accuracy]] should not be the measure of forecast class probabilities. [a_priori] 2026-11-13 16:21 ST82 status by AC1: H75 -> supported, evidence H45, H44: Accuracy uses a forecast probability only through which side of a threshold it falls, and counts every misclassificatio… H19 [supported] [[TM22:AUC]] should not be the measure of forecast class probabilities. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM22 term by AC1: AUC [variable] 2026-11-13 13:20 H19 hypothesis by AC1: [[TM22:AUC]] should not be the measure of forecast class probabilities: it measures how well cases are ranked, not whether the probabilities are right, and it… [a_priori] 2026-11-13 13:31 ST19 status by AC1: H19 -> supported, evidence NT8: Hand (2009, NT8): the AUC averages the misclassification loss over a cost distribution that depends on the classifier,… 2026-11-13 16:18 H49 hypothesis by AC1: The [[TM22:AUC]] is equivalent to averaging the misclassification loss over a distribution of misclassification costs that depends on the classifier. [a_priori] 2026-11-13 16:21 H76 hypothesis by AC1: [[TM22:AUC]] should not be the measure of forecast class probabilities. [a_priori] 2026-11-13 16:21 ST83 status by AC1: H76 -> supported, evidence H49: The AUC averages the misclassification loss over a cost distribution that depends on the classifier, so different class… H20 [supported] With rare positive cases, the [[TM23:precision-recall curve]] should be reported rather than the ROC curve alone. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM23 term by AC1: precision-recall curve [concept] 2026-11-13 13:20 H20 hypothesis by AC1: With rare positive cases, the [[TM23:precision-recall curve]] should be reported rather than the ROC curve alone, because the ROC curve can look good while mos… [a_priori] 2026-11-13 13:31 ST20 status by AC1: H20 -> supported, evidence SR8: Saito and Rehmsmeier (2015, SR8, title and abstract): 'The Precision-Recall Plot Is More Informative than the ROC Plot… 2026-11-13 16:18 H50 hypothesis by AC1: On imbalanced data, the [[TM23:precision-recall curve]] is more informative than the ROC curve for evaluating binary classifiers. [a_priori] 2026-11-13 16:21 H77 hypothesis by AC1: With rare positive cases, the [[TM23:precision-recall curve]] should be reported rather than the ROC curve alone. [a_priori] 2026-11-13 16:21 ST84 status by AC1: H77 -> supported, evidence H50: On imbalanced data the ROC curve can be deceptive about how reliable a classifier is, and the precision-recall curve is… H21 [supported] When the compared methods target different functionals, one the mean and another the median, each should be evaluated with a measure consistent for its own target, or the methods made to target the same functional. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:20 H21 hypothesis by AC1: When the compared methods target different functionals (one the mean, another the median), each should be evaluated with a measure consistent for its own targe… [a_priori] 2026-11-13 13:31 ST21 status by AC1: H21 -> supported, evidence SR1, NT2: Kolassa (2020, NT2): different error measures are minimised by different summaries of the same predictive density, so n… 2026-11-13 16:16 H37 hypothesis by AC1: Which point forecast has the lowest expected score depends on the scoring function: a [[TM1:consistent scoring function]] for a functional, such as the mean or… [a_priori] 2026-11-13 16:21 H78 hypothesis by AC1: When the compared methods target different functionals, one the mean and another the median, each should be evaluated with a measure consistent for its own tar… [a_priori] 2026-11-13 16:21 ST85 status by AC1: H78 -> supported, evidence H37: Each scoring function is lowest in expectation for the functional it is consistent for, so one measure favours the meth… H22 [supported] When methods are compared over many series, their accuracy should be summarised by the mean, the median and the quartiles of a scaled measure across the series, such as [[TM6:RMSSE]] or [[TM8:MASE]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM6 term by AC1: RMSSE [variable] 2026-11-13 13:18 TM8 term by AC1: MASE [variable] 2026-11-13 13:20 H22 hypothesis by AC1: When methods are compared over many series or data sets, the distribution of their scaled errors across the series should be reported, not only the mean, becau… [a_priori] 2026-11-13 13:31 ST22 status by AC1: H22 -> supported, evidence SR4: Demsar (2006, SR4, Section 3.1): 'Averages are also susceptible to outliers. They allow classifier's excellent performa… 2026-11-13 14:07 H31 hypothesis by AC1: When methods are compared over many series, their accuracy should be summarised by aggregates of a scaled measure across the series, the mean together with the… [a_priori] 2026-11-13 14:08 ST39 status by AC1: H31 -> supported, evidence SR4, SR7, SR9: Demsar (2006, SR4, Section 3.1): 'Averages are also susceptible to outliers'. Hyndman and Koehler (2006, SR7, Section 3… 2026-11-13 16:18 H53 hypothesis by AC1: Scaled errors, as in [[TM8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. [a_priori] 2026-11-13 16:19 H57 hypothesis by AC1: Averages of a performance measure over many data sets are susceptible to outliers. [a_priori] 2026-11-13 16:22 H84 hypothesis by AC1: When methods are compared over many series, their accuracy should be summarised by the mean, the median and the quartiles of a scaled measure across the series… [a_priori] 2026-11-13 16:22 ST91 status by AC1: H84 -> supported, evidence H53, H57: A scaled measure can be averaged over series, but averages over many data sets are susceptible to outliers; the median… H23 [supported] To tell whether several methods whose aggregate accuracy is similar differ statistically significantly over many series or data sets, their ranks on each series should be tested with the [[TM25:Friedman test]] followed by [[TM26:Nemenyi test]] post-hoc comparisons. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM25 term by AC1: Friedman test [method] 2026-11-13 13:19 TM26 term by AC1: Nemenyi test [method] 2026-11-13 13:20 H23 hypothesis by AC1: When several methods are compared over many data sets, their differences should be tested with the [[TM25:Friedman test]] on ranks followed by [[TM26:Nemenyi t… [a_priori] 2026-11-13 13:31 ST23 status by AC1: H23 -> supported, evidence SR4: Demsar (2006, SR4, abstract): 'we recommend a set of simple, yet safe and robust non-parametric tests', 'the Friedman t… 2026-11-13 14:07 H32 hypothesis by AC1: To tell whether several methods whose aggregate accuracy is similar differ significantly over many series or data sets, their ranks on each series should be te… [a_priori] 2026-11-13 14:08 ST40 status by AC1: H32 -> supported, evidence SR4: Demsar (2006, SR4, abstract; Sections 3.1 and 3.2.2): it recommends 'the Friedman test with the corresponding post-hoc… 2026-11-13 16:19 H58 hypothesis by AC1: The [[TM25:Friedman test]], followed by the [[TM26:Nemenyi test]], compares several methods over many data sets by their ranks on each data set, assuming only… [a_priori] 2026-11-13 16:22 H85 hypothesis by AC1: To tell whether several methods whose aggregate accuracy is similar differ significantly over many series or data sets, their ranks on each series should be te… [a_priori] 2026-11-13 16:22 ST92 status by AC1: H85 -> supported, evidence H58: The Friedman test, followed by the Nemenyi test, compares methods by their ranks on each data set and assumes only limi… 2026-11-13 16:25 H91 hypothesis by AC1: To tell whether several methods whose aggregate accuracy is similar differ statistically significantly over many series or data sets, their ranks on each serie… [a_priori] 2026-11-13 16:25 ST98 status by AC1: H91 -> supported, evidence H58: The Friedman test, followed by the Nemenyi test, compares methods by their ranks on each data set and assumes only limi… H24 [supported] To tell which methods differ statistically significantly from the best one over many series, multiple comparisons with the best ([[TM27:MCB]]) on ranks should be used. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM27 term by AC1: MCB [method] 2026-11-13 13:20 H24 hypothesis by AC1: When the question is which methods differ from the best one over many series, multiple comparisons with the best ([[TM27:MCB]]) on ranks answer it, as in the s… [a_priori] 2026-11-13 13:31 ST24 status by AC1: H24 -> supported, evidence NT9: Koning et al. (2005, NT9) test the M3 results with non-parametric multiple comparisons on ranks over series, including… 2026-11-13 14:07 H33 hypothesis by AC1: To tell which methods differ significantly from the best one over many series, multiple comparisons with the best ([[TM27:MCB]]) on ranks should be used, as in… [a_priori] 2026-11-13 14:08 ST41 status by AC1: H33 -> supported, evidence NT9: Koning et al. (2005, NT9) test the M3 results with multiple comparisons with the best (MCB) on ranks over series, which… 2026-11-13 16:19 H59 hypothesis by AC1: [[TM27:MCB]] on ranks was used to test the differences between the methods of the M3 competition. [a_priori] 2026-11-13 16:22 H86 hypothesis by AC1: To tell which methods differ significantly from the best one over many series, multiple comparisons with the best ([[TM27:MCB]]) on ranks should be used. [a_priori] 2026-11-13 16:22 ST93 status by AC1: H86 -> supported, evidence H59: Multiple comparisons with the best on ranks test which methods differ from the best one, and were used for this on the… 2026-11-13 16:25 H92 hypothesis by AC1: To tell which methods differ statistically significantly from the best one over many series, multiple comparisons with the best ([[TM27:MCB]]) on ranks should… [a_priori] 2026-11-13 16:25 ST99 status by AC1: H92 -> supported, evidence H59: Multiple comparisons with the best on ranks test which methods differ from the best one, and were used for this on the… H25 [supported] The accuracy of two forecasts of the same series should be compared with the [[TM28:Diebold-Mariano test]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM28 term by AC1: Diebold-Mariano test [method] 2026-11-13 13:20 H25 hypothesis by AC1: The accuracy of two forecasts of the same series should be compared with the [[TM28:Diebold-Mariano test]], which allows forecast errors that are serially corr… [a_priori] 2026-11-13 13:31 ST25 status by AC1: H25 -> supported, evidence NT10: Diebold and Mariano (1995, NT10) propose tests of no difference in the accuracy of two competing forecasts that allow n… 2026-11-13 16:19 H60 hypothesis by AC1: The [[TM28:Diebold-Mariano test]] of equal accuracy of two forecasts allows forecast errors that are serially correlated and not normally distributed. [a_priori] 2026-11-13 16:21 H79 hypothesis by AC1: The accuracy of two forecasts of the same series should be compared with the [[TM28:Diebold-Mariano test]]. [a_priori] 2026-11-13 16:21 ST86 status by AC1: H79 -> supported, evidence H60: The Diebold-Mariano test allows forecast errors that are serially correlated and not normally distributed, as the error… H26 [supported] That a competition or a well-known study used an error measure should not be the reason for choosing it. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:20 H26 hypothesis by AC1: An error measure should not be chosen because a competition or a well-known study used it: the measure has to be consistent for what the compared methods predi… [a_priori] 2026-11-13 13:31 ST26 status by AC1: H26 -> supported, evidence SR1, NT2: Kolassa (2020, NT2), a commentary on the measures of the M4 competition: the best point forecast depends on the error m… 2026-11-13 16:16 H37 hypothesis by AC1: Which point forecast has the lowest expected score depends on the scoring function: a [[TM1:consistent scoring function]] for a functional, such as the mean or… [a_priori] 2026-11-13 16:21 H80 hypothesis by AC1: That a competition or a well-known study used an error measure should not be the reason for choosing it. [a_priori] 2026-11-13 16:21 ST87 status by AC1: H80 -> supported, evidence H37: A measure fits a study when it is consistent for what the compared methods predict; that a competition used a measure s… H28 [supported] Scale-dependent error measures, such as [[TM5:RMSE]] and [[TM7:MAE]], should not be summed or averaged across series on different scales. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM5 term by AC1: RMSE [variable] 2026-11-13 13:18 TM7 term by AC1: MAE [variable] 2026-11-13 13:45 H28 hypothesis by AC1: Scale-dependent error measures ([[TM5:RMSE]], [[TM7:MAE]]) should not be summed or averaged across series on different scales: the series with the largest valu… [a_priori] 2026-11-13 13:51 ST31 status by AC1: H28 -> supported, evidence SR7, NT12: Hyndman and Koehler (2006, SR7, Section 2.1): scale-dependent measures 'are useful when comparing different methods on… 2026-11-13 13:51 ST33 status by AC1: H28 -> supported, evidence SR7, NT12: Hyndman and Koehler (2006, SR7, Section 2.1): scale-dependent measures 'are useful when comparing different methods on… 2026-11-13 16:18 H51 hypothesis by AC1: Scale-dependent error measures, such as [[TM5:RMSE]] and [[TM7:MAE]], cannot be compared across series on different scales. [a_priori] 2026-11-13 16:18 H52 hypothesis by AC1: When the errors of series on different scales are summed or averaged without scaling, the series with the largest values dominate the result. [a_priori] 2026-11-13 16:21 H81 hypothesis by AC1: Scale-dependent error measures, such as [[TM5:RMSE]] and [[TM7:MAE]], should not be summed or averaged across series on different scales. [a_priori] 2026-11-13 16:21 ST88 status by AC1: H81 -> supported, evidence H51, H52: Scale-dependent measures cannot be compared across series on different scales, and summed or averaged without scaling t… H29 [supported] To aggregate the accuracy of point forecasts over series on different scales, each series' errors should first be scaled by a quantity of that series that does not depend on the compared forecasts, such as the in-sample error of its one-step naive forecast in [[TM8:MASE]] or [[TM6:RMSSE]], before t… 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM6 term by AC1: RMSSE [variable] 2026-11-13 13:18 TM8 term by AC1: MASE [variable] 2026-11-13 13:45 H29 hypothesis by AC1: To aggregate the accuracy of point forecasts over series on different scales, each series' errors should first be scaled by a quantity of that series (as in [[… [a_priori] 2026-11-13 13:51 ST32 status by AC1: H29 -> supported, evidence SR7, NT3: Hyndman and Koehler (2006, SR7, Section 3): 'measures based on scaled errors should become the standard approach in com… 2026-11-13 14:08 ST42 status by AC1: H29 -> supported, evidence SR7, SR9, NT3: Hyndman and Koehler (2006, SR7, Section 3): 'measures based on scaled errors should become the standard approach in com… 2026-11-13 16:18 H51 hypothesis by AC1: Scale-dependent error measures, such as [[TM5:RMSE]] and [[TM7:MAE]], cannot be compared across series on different scales. [a_priori] 2026-11-13 16:18 H53 hypothesis by AC1: Scaled errors, as in [[TM8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. [a_priori] 2026-11-13 16:18 H54 hypothesis by AC1: The M5 accuracy competition scored point forecasts with the [[TM6:RMSSE]]: each series' squared errors divided by the in-sample mean squared error of its one-s… [a_priori] 2026-11-13 16:21 H82 hypothesis by AC1: To aggregate the accuracy of point forecasts over series on different scales, each series' errors should first be scaled by a quantity of that series that does… [a_priori] 2026-11-13 16:21 ST89 status by AC1: H82 -> supported, evidence H51, H53, H54: Scaled errors are independent of the scale of the data and can be averaged over series; dividing by a quantity of the s… H34 [supported] When quantile forecasts are compared over series on different scales, each series' [[TM12:pinball loss]] should be scaled by a quantity of that series. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM12 term by AC1: pinball loss [variable] 2026-11-13 14:07 H34 hypothesis by AC1: When quantile forecasts are compared over series on different scales, each series' [[TM12:pinball loss]] should be scaled by a quantity of that series, as the… [a_priori] 2026-11-13 14:08 ST37 status by AC1: H34 -> supported, evidence SR9, NT4: The M5 Competitors' Guide (SR9, Probabilistic forecasts): forecasts of quantiles are evaluated with the 'Scaled Pinball… 2026-11-13 16:17 H41 hypothesis by AC1: The [[TM12:pinball loss]] at a level is a [[TM1:consistent scoring function]] for the quantile at that level. [a_priori] 2026-11-13 16:18 H51 hypothesis by AC1: Scale-dependent error measures, such as [[TM5:RMSE]] and [[TM7:MAE]], cannot be compared across series on different scales. [a_priori] 2026-11-13 16:18 H55 hypothesis by AC1: The M5 uncertainty competition scored quantile forecasts with the scaled pinball loss: each series' [[TM12:pinball loss]] divided by the in-sample mean absolut… [a_priori] 2026-11-13 16:22 H87 hypothesis by AC1: When quantile forecasts are compared over series on different scales, each series' [[TM12:pinball loss]] should be scaled by a quantity of that series. [a_priori] 2026-11-13 16:22 ST94 status by AC1: H87 -> supported, evidence H55, H41, H51: Pinball losses of series on different scales cannot be compared unscaled; dividing each series' pinball loss by a quant… H35 [supported] When prediction intervals are compared over series on different scales, each series' [[TM13:interval score]] should be scaled by a quantity of that series. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM13 term by AC1: interval score [variable] 2026-11-13 14:08 H35 hypothesis by AC1: When prediction intervals are compared over series on different scales, each series' [[TM13:interval score]] should be scaled by a quantity of that series, as… [a_priori] 2026-11-13 14:08 ST38 status by AC1: H35 -> supported, evidence NT13, SR7, SR2: The M4 competition (NT13) evaluated prediction intervals with the mean scaled interval score, the interval score of Gne… 2026-11-13 16:17 H42 hypothesis by AC1: The [[TM13:interval score]] is a [[TM2:proper scoring rule]] for central prediction intervals that addresses their width as well as their coverage. [a_priori] 2026-11-13 16:18 H51 hypothesis by AC1: Scale-dependent error measures, such as [[TM5:RMSE]] and [[TM7:MAE]], cannot be compared across series on different scales. [a_priori] 2026-11-13 16:19 H56 hypothesis by AC1: The M4 competition scored prediction intervals with the mean scaled interval score: the [[TM13:interval score]] of each series scaled by an in-sample error of… [a_priori] 2026-11-13 16:22 H88 hypothesis by AC1: When prediction intervals are compared over series on different scales, each series' [[TM13:interval score]] should be scaled by a quantity of that series. [a_priori] 2026-11-13 16:22 ST95 status by AC1: H88 -> supported, evidence H56, H42, H51: Interval scores of series on different scales cannot be compared unscaled; dividing each series' interval score by a po… H36 [supported] Squared error is a [[TM1:consistent scoring function]] for the mean. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM1 term by AC1: consistent scoring function [concept] 2026-11-13 16:16 H36 hypothesis by AC1: Squared error is a [[TM1:consistent scoring function]] for the mean. [a_priori] 2026-11-13 16:16 ST43 status by AC1: H36 -> supported, evidence XT1, SR1: Gneiting (2011), Section 5 and Theorem 3.1. H37 [supported] Which point forecast has the lowest expected score depends on the scoring function: a [[TM1:consistent scoring function]] for a functional, such as the mean or a quantile, is lowest in expectation for the forecast that equals that functional of the outcome's distribution. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM1 term by AC1: consistent scoring function [concept] 2026-11-13 16:16 H37 hypothesis by AC1: Which point forecast has the lowest expected score depends on the scoring function: a [[TM1:consistent scoring function]] for a functional, such as the mean or… [a_priori] 2026-11-13 16:16 ST44 status by AC1: H37 -> supported, evidence XT2, SR1, NT2, NT14: Gneiting (2011, abstract) defines consistency and states that the scoring function must be consistent for the functiona… H38 [supported] Absolute error is a [[TM1:consistent scoring function]] for the median. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM1 term by AC1: consistent scoring function [concept] 2026-11-13 16:16 H38 hypothesis by AC1: Absolute error is a [[TM1:consistent scoring function]] for the median. [a_priori] 2026-11-13 16:16 ST45 status by AC1: H38 -> supported, evidence XT3, SR1: Gneiting (2011), Section 5. H39 [supported] The absolute percentage error averaged by [[TM9:MAPE]] is consistent for the [[TM11:median of order -1]], a functional that supports severe underforecasts compared with the mean or the median. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM9 term by AC1: MAPE [variable] 2026-11-13 13:19 TM11 term by AC1: median of order -1 [concept] 2026-11-13 16:16 H39 hypothesis by AC1: The absolute percentage error averaged by [[TM9:MAPE]] is consistent for the [[TM11:median of order -1]], a functional that supports severe underforecasts comp… [a_priori] 2026-11-13 16:17 ST46 status by AC1: H39 -> supported, evidence XT4, XT5, SR1: Gneiting (2011), Section 5. H40 [supported] For the same outcome, [[TM10:sMAPE]] penalises a forecast below the outcome more heavily than a forecast above it. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM10 term by AC1: sMAPE [variable] 2026-11-13 16:17 H40 hypothesis by AC1: For the same outcome, [[TM10:sMAPE]] penalises a forecast below the outcome more heavily than a forecast above it. [a_priori] 2026-11-13 16:17 ST47 status by AC1: H40 -> supported, evidence XT6, SR7, NT5, NT14: Hyndman and Koehler (2006), Section 2.2; Goodwin and Lawton (1999), read in full by the guarantor (NT14). H41 [supported] The [[TM12:pinball loss]] at a level is a [[TM1:consistent scoring function]] for the quantile at that level. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM1 term by AC1: consistent scoring function [concept] 2026-11-13 13:19 TM12 term by AC1: pinball loss [variable] 2026-11-13 16:17 H41 hypothesis by AC1: The [[TM12:pinball loss]] at a level is a [[TM1:consistent scoring function]] for the quantile at that level. [a_priori] 2026-11-13 16:17 ST48 status by AC1: H41 -> supported, evidence XT7, SR1: Gneiting (2011), abstract and Theorem 3.3: the scoring functions consistent for a quantile are the generalized piecewis… H42 [supported] The [[TM13:interval score]] is a [[TM2:proper scoring rule]] for central prediction intervals that addresses both their width and their coverage. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM2 term by AC1: proper scoring rule [concept] 2026-11-13 13:19 TM13 term by AC1: interval score [variable] 2026-11-13 16:17 H42 hypothesis by AC1: The [[TM13:interval score]] is a [[TM2:proper scoring rule]] for central prediction intervals that addresses their width as well as their coverage. [a_priori] 2026-11-13 16:17 ST49 status by AC1: H42 -> supported, evidence XT8, XT9, SR2: Gneiting and Raftery (2007), abstract, Section 1 and Section 6.2. 2026-11-13 16:25 H89 hypothesis by AC1: The [[TM13:interval score]] is a [[TM2:proper scoring rule]] for central prediction intervals that addresses both their width and their coverage. [a_priori] 2026-11-13 16:25 ST96 status by AC1: H89 -> supported, evidence XT8, XT9, SR2: Gneiting and Raftery (2007), abstract, Section 1 and Section 6.2. H43 [supported] The [[TM15:CRPS]] is a [[TM2:proper scoring rule]] for predictive distributions of continuous outcomes. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM2 term by AC1: proper scoring rule [concept] 2026-11-13 13:19 TM15 term by AC1: CRPS [variable] 2026-11-13 16:17 H43 hypothesis by AC1: The [[TM15:CRPS]] is a [[TM2:proper scoring rule]] for predictive distributions of continuous outcomes. [a_priori] 2026-11-13 16:17 ST50 status by AC1: H43 -> supported, evidence XT10, SR2: Gneiting and Raftery (2007), Section 4.2. H44 [supported] The [[TM16:log score]] is a strictly [[TM2:proper scoring rule]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM2 term by AC1: proper scoring rule [concept] 2026-11-13 13:19 TM16 term by AC1: log score [variable] 2026-11-13 16:17 H44 hypothesis by AC1: The [[TM16:log score]] is a strictly [[TM2:proper scoring rule]]. [a_priori] 2026-11-13 16:17 ST51 status by AC1: H44 -> supported, evidence XT11, SR2: Gneiting and Raftery (2007), Section 4.1. H45 [supported] The [[TM17:Brier score]] is a [[TM2:proper scoring rule]] for forecast probabilities of classes. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM2 term by AC1: proper scoring rule [concept] 2026-11-13 13:19 TM17 term by AC1: Brier score [variable] 2026-11-13 16:17 H45 hypothesis by AC1: The [[TM17:Brier score]] is a [[TM2:proper scoring rule]] for forecast probabilities of classes. [a_priori] 2026-11-13 16:17 ST52 status by AC1: H45 -> supported, evidence XT12, SR2, NT7: Gneiting and Raftery (2007), Section 3, Example 1; Brier (1950), checked by its abstract and bibliographic record only… H46 [supported] Predictive distributions are assessed by maximising their [[TM20:sharpness]] subject to their [[TM19:calibration]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM19 term by AC1: calibration [concept] 2026-11-13 13:19 TM20 term by AC1: sharpness [concept] 2026-11-13 16:17 H46 hypothesis by AC1: Predictive distributions are assessed by maximising their [[TM20:sharpness]] subject to their [[TM19:calibration]]. [a_priori] 2026-11-13 16:18 ST53 status by AC1: H46 -> supported, evidence XT13, SR3: Gneiting, Balabdaoui and Raftery (2007), abstract and Sections 3.1 and 3.3. H47 [supported] The calibration of predictive distributions of counts can be assessed with a randomised [[TM18:probability integral transform]]. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM18 term by AC1: probability integral transform [concept] 2026-11-13 16:18 H47 hypothesis by AC1: The calibration of predictive distributions of counts can be assessed with a randomised [[TM18:probability integral transform]]. [a_priori] 2026-11-13 16:18 ST54 status by AC1: H47 -> supported, evidence NT6, NT14: Kolassa (2016), read in full by the guarantor (NT14). H48 [supported] On [[TM24:intermittent series]], absolute-error measures give their best scores to forecasts close to zero. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM24 term by AC1: intermittent series [data] 2026-11-13 16:18 H48 hypothesis by AC1: On [[TM24:intermittent series]], absolute-error measures give their best scores to forecasts close to zero. [a_priori] 2026-11-13 16:18 ST55 status by AC1: H48 -> supported, evidence XT14, SR9, NT6, NT14: The M5 competitors' guide (Point forecasts); Kolassa (2016), read in full by the guarantor (NT14). H49 [supported] The [[TM22:AUC]] is equivalent to averaging the misclassification loss over a distribution of misclassification costs that depends on the classifier. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM22 term by AC1: AUC [variable] 2026-11-13 16:18 H49 hypothesis by AC1: The [[TM22:AUC]] is equivalent to averaging the misclassification loss over a distribution of misclassification costs that depends on the classifier. [a_priori] 2026-11-13 16:18 ST56 status by AC1: H49 -> supported, evidence NT8: Hand (2009), checked by its abstract only (NT8). H50 [supported] On imbalanced data, the [[TM23:precision-recall curve]] is more informative than the ROC curve for evaluating binary classifiers. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM23 term by AC1: precision-recall curve [concept] 2026-11-13 16:18 H50 hypothesis by AC1: On imbalanced data, the [[TM23:precision-recall curve]] is more informative than the ROC curve for evaluating binary classifiers. [a_priori] 2026-11-13 16:18 ST57 status by AC1: H50 -> supported, evidence XT15, XT16, SR8: Saito and Rehmsmeier (2015), title and abstract. H51 [supported] Scale-dependent error measures, such as [[TM5:RMSE]] and [[TM7:MAE]], cannot be compared across series on different scales. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM5 term by AC1: RMSE [variable] 2026-11-13 13:18 TM7 term by AC1: MAE [variable] 2026-11-13 16:18 H51 hypothesis by AC1: Scale-dependent error measures, such as [[TM5:RMSE]] and [[TM7:MAE]], cannot be compared across series on different scales. [a_priori] 2026-11-13 16:18 ST58 status by AC1: H51 -> supported, evidence XT17, XT18, SR7: Hyndman and Koehler (2006), Sections 2.1 and 2.4. H52 [supported] When the errors of series on different scales are summed or averaged without scaling, the series with the largest values dominate the result. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 16:18 H52 hypothesis by AC1: When the errors of series on different scales are summed or averaged without scaling, the series with the largest values dominate the result. [a_priori] 2026-11-13 16:18 ST59 status by AC1: H52 -> supported, evidence NT12, NT14: Athanasopoulos and Kourentzes (2023), read in full by the guarantor (NT14). H53 [supported] Scaled errors, as in [[TM8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM8 term by AC1: MASE [variable] 2026-11-13 16:18 H53 hypothesis by AC1: Scaled errors, as in [[TM8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. [a_priori] 2026-11-13 16:18 ST60 status by AC1: H53 -> supported, evidence XT19, SR7: Hyndman and Koehler (2006), abstract and Section 3. H54 [supported] The M5 accuracy competition scored point forecasts with the [[TM6:RMSSE]]: each series' squared errors divided by the in-sample mean squared error of its one-step naive forecast. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM6 term by AC1: RMSSE [variable] 2026-11-13 16:18 H54 hypothesis by AC1: The M5 accuracy competition scored point forecasts with the [[TM6:RMSSE]]: each series' squared errors divided by the in-sample mean squared error of its one-s… [a_priori] 2026-11-13 16:18 ST61 status by AC1: H54 -> supported, evidence XT20, XT21, SR9, NT3, NT14: The M5 competitors' guide (Point forecasts); the M5 accuracy paper, read in full by the guarantor (NT14). H55 [supported] The M5 uncertainty competition scored quantile forecasts with the scaled pinball loss: each series' [[TM12:pinball loss]] divided by the in-sample mean absolute error of its one-step naive forecast. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM12 term by AC1: pinball loss [variable] 2026-11-13 16:18 H55 hypothesis by AC1: The M5 uncertainty competition scored quantile forecasts with the scaled pinball loss: each series' [[TM12:pinball loss]] divided by the in-sample mean absolut… [a_priori] 2026-11-13 16:18 ST62 status by AC1: H55 -> supported, evidence XT22, SR9, NT4, NT14: The M5 competitors' guide (Probabilistic forecasts); the M5 uncertainty paper, read in full by the guarantor (NT14). H56 [supported] The M4 competition scored prediction intervals with the mean scaled interval score: the [[TM13:interval score]] of each series scaled by an in-sample error of that series. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM13 term by AC1: interval score [variable] 2026-11-13 16:19 H56 hypothesis by AC1: The M4 competition scored prediction intervals with the mean scaled interval score: the [[TM13:interval score]] of each series scaled by an in-sample error of… [a_priori] 2026-11-13 16:19 ST63 status by AC1: H56 -> supported, evidence NT13, NT14: The M4 paper, read in full by the guarantor (NT14). H57 [supported] Averages of a performance measure over many data sets are susceptible to outliers. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 16:19 H57 hypothesis by AC1: Averages of a performance measure over many data sets are susceptible to outliers. [a_priori] 2026-11-13 16:19 ST64 status by AC1: H57 -> supported, evidence XT23, SR4: Demsar (2006), Section 3.1. H58 [supported] The [[TM25:Friedman test]], followed by the [[TM26:Nemenyi test]], compares several methods over many data sets by their ranks on each data set, assuming only limited commensurability of the measure across data sets. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM25 term by AC1: Friedman test [method] 2026-11-13 13:19 TM26 term by AC1: Nemenyi test [method] 2026-11-13 16:19 H58 hypothesis by AC1: The [[TM25:Friedman test]], followed by the [[TM26:Nemenyi test]], compares several methods over many data sets by their ranks on each data set, assuming only… [a_priori] 2026-11-13 16:19 ST65 status by AC1: H58 -> supported, evidence XT24, XT25, XT26, SR4: Demsar (2006), Sections 3.2.2 and 5. H59 [supported] [[TM27:MCB]] on ranks was used to test the differences between the methods of the M3 competition. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM27 term by AC1: MCB [method] 2026-11-13 16:19 H59 hypothesis by AC1: [[TM27:MCB]] on ranks was used to test the differences between the methods of the M3 competition. [a_priori] 2026-11-13 16:19 ST66 status by AC1: H59 -> supported, evidence NT9, NT14: Koning, Franses, Hibon and Stekler (2005), read in full by the guarantor (NT14). H60 [supported] The [[TM28:Diebold-Mariano test]] of equal accuracy of two forecasts allows forecast errors that are serially correlated and not normally distributed. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:19 TM28 term by AC1: Diebold-Mariano test [method] 2026-11-13 16:19 H60 hypothesis by AC1: The [[TM28:Diebold-Mariano test]] of equal accuracy of two forecasts allows forecast errors that are serially correlated and not normally distributed. [a_priori] 2026-11-13 16:19 ST67 status by AC1: H60 -> supported, evidence NT10, NT14: Diebold and Mariano (1995), read in full by the guarantor (NT14). TIMELINE (every event but results, in log order: seq, time, ID, type, writer) 1 2026-11-13 13:16 INIT init by -: began "Good practice for evaluating predictions: what the literature supports" under lab record v0.22 2 2026-11-13 13:16 AC1 actor by AC1: registers Claude Code (agent) 3 2026-11-13 13:16 AC2 actor by AC1: registers Ivan Svetunkov (human, ORCID 0000-0001-7826-0281) 4 2026-11-13 13:16 NT1 note by AC1: Brief requirements (from the supervisor, 2026-11-13): (1) a claimset based on prediction.md of entrylib/study-planning (v0.2, commit 104ba1c); (2) a pure liter… 5 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 6 2026-11-13 13:18 TM1 term by AC1: consistent scoring function [concept] 7 2026-11-13 13:18 TM2 term by AC1: proper scoring rule [concept] 8 2026-11-13 13:18 TM3 term by AC1: point forecast of the mean [concept] 9 2026-11-13 13:18 TM4 term by AC1: point forecast of the median [concept] 10 2026-11-13 13:18 TM5 term by AC1: RMSE [variable] 11 2026-11-13 13:18 TM6 term by AC1: RMSSE [variable] 12 2026-11-13 13:18 TM7 term by AC1: MAE [variable] 13 2026-11-13 13:18 TM8 term by AC1: MASE [variable] 14 2026-11-13 13:18 TM9 term by AC1: MAPE [variable] 15 2026-11-13 13:18 TM10 term by AC1: sMAPE [variable] 16 2026-11-13 13:19 TM11 term by AC1: median of order -1 [concept] 17 2026-11-13 13:19 TM12 term by AC1: pinball loss [variable] 18 2026-11-13 13:19 TM13 term by AC1: interval score [variable] 19 2026-11-13 13:19 TM14 term by AC1: coverage [variable] 20 2026-11-13 13:19 TM15 term by AC1: CRPS [variable] 21 2026-11-13 13:19 TM16 term by AC1: log score [variable] 22 2026-11-13 13:19 TM17 term by AC1: Brier score [variable] 23 2026-11-13 13:19 TM18 term by AC1: probability integral transform [concept] 24 2026-11-13 13:19 TM19 term by AC1: calibration [concept] 25 2026-11-13 13:19 TM20 term by AC1: sharpness [concept] 26 2026-11-13 13:19 TM21 term by AC1: accuracy [variable] 27 2026-11-13 13:19 TM22 term by AC1: AUC [variable] 28 2026-11-13 13:19 TM23 term by AC1: precision-recall curve [concept] 29 2026-11-13 13:19 TM24 term by AC1: intermittent series [data] 30 2026-11-13 13:19 TM25 term by AC1: Friedman test [method] 31 2026-11-13 13:19 TM26 term by AC1: Nemenyi test [method] 32 2026-11-13 13:19 TM27 term by AC1: MCB [method] 33 2026-11-13 13:19 TM28 term by AC1: Diebold-Mariano test [method] 34 2026-11-13 13:19 TM29 term by AC1: bias adjustment [method] 35 2026-11-13 13:19 TM30 term by AC1: ETS [method] 36 2026-11-13 13:20 H1 hypothesis by AC1: To evaluate point forecasts, use a [[TM1:consistent scoring function]] for the functional the forecasts target (the mean, the median or a quantile): a scoring… [a_priori] 37 2026-11-13 13:20 H2 hypothesis by AC1: Point forecasts of the mean ([[TM3:point forecast of the mean]]) should be evaluated with squared error ([[TM5:RMSE]]), the scoring function consistent for the… [a_priori] 38 2026-11-13 13:20 H3 hypothesis by AC1: When [[TM3:point forecasts of the mean]] are compared over series on different scales, the [[TM5:RMSE]] of each series should be scaled by the in-sample RMSE o… [a_priori] 39 2026-11-13 13:20 H4 hypothesis by AC1: The point forecasts of [[TM30:ETS]] models are [[TM3:point forecasts of the mean]] (conditional means of the forecast distribution), except, beyond one step ah… [a_priori] 40 2026-11-13 13:20 H5 hypothesis by AC1: Point forecasts made on a Box-Cox or log scale and transformed back to the original scale are [[TM4:point forecasts of the median]], not of the mean, unless th… [a_priori] 41 2026-11-13 13:20 H6 hypothesis by AC1: [[TM7:MAE]] and [[TM8:MASE]] should not be the main measure for [[TM3:point forecasts of the mean]]: absolute error is consistent for the median and rewards me… [a_priori] 42 2026-11-13 13:20 H7 hypothesis by AC1: [[TM9:MAPE]] should not be the measure for point forecasts of the mean or the median: it is consistent for the [[TM11:median of order -1]], which lies below th… [a_priori] 43 2026-11-13 13:20 H8 hypothesis by AC1: [[TM10:sMAPE]] should not be the measure for point forecasts: it penalises errors of the same size differently depending on their sign, and it is undefined whe… [a_priori] 44 2026-11-13 13:20 H9 hypothesis by AC1: Point forecasts of the median ([[TM4:point forecast of the median]]) should be evaluated with absolute error ([[TM7:MAE]]), the scoring function consistent for… [a_priori] 45 2026-11-13 13:20 H10 hypothesis by AC1: On [[TM24:intermittent series]], absolute-error measures should not be used alone: the median of the outcome can be zero, and absolute error then rewards a for… [a_priori] 46 2026-11-13 13:20 H11 hypothesis by AC1: Quantile forecasts should be evaluated with the [[TM12:pinball loss]] at the quantile's level, the scoring function consistent for that quantile, averaged over… [a_priori] 47 2026-11-13 13:20 H12 hypothesis by AC1: Prediction intervals should be evaluated with the [[TM13:interval score]], a proper scoring rule for central intervals, reported together with their [[TM14:cov… [a_priori] 48 2026-11-13 13:20 H13 hypothesis by AC1: [[TM14:coverage]] alone should not be the measure of prediction intervals: it does not measure their width, so sharp intervals and needlessly wide ones can rea… [a_priori] 49 2026-11-13 13:20 H14 hypothesis by AC1: Predictive distributions should be evaluated with a [[TM2:proper scoring rule]], such as the [[TM15:CRPS]] or the [[TM16:log score]]. [a_priori] 50 2026-11-13 13:20 H15 hypothesis by AC1: The evaluation of predictive distributions should check their [[TM19:calibration]], with the [[TM18:probability integral transform]] or coverage at several lev… [a_priori] 51 2026-11-13 13:20 H16 hypothesis by AC1: Predictive distributions of counts, such as forecasts of intermittent sales, should be evaluated with proper scoring rules and a randomised [[TM18:probability… [a_priori] 52 2026-11-13 13:20 H17 hypothesis by AC1: Forecast probabilities of classes should be evaluated with a [[TM2:proper scoring rule]], such as the [[TM16:log score]] or the [[TM17:Brier score]]. [a_priori] 53 2026-11-13 13:20 H18 hypothesis by AC1: [[TM21:accuracy]] should not be the measure of forecast class probabilities: it ignores the probabilities beyond which side of a threshold they fall, and weigh… [a_priori] 54 2026-11-13 13:20 H19 hypothesis by AC1: [[TM22:AUC]] should not be the measure of forecast class probabilities: it measures how well cases are ranked, not whether the probabilities are right, and it… [a_priori] 55 2026-11-13 13:20 H20 hypothesis by AC1: With rare positive cases, the [[TM23:precision-recall curve]] should be reported rather than the ROC curve alone, because the ROC curve can look good while mos… [a_priori] 56 2026-11-13 13:20 H21 hypothesis by AC1: When the compared methods target different functionals (one the mean, another the median), each should be evaluated with a measure consistent for its own targe… [a_priori] 57 2026-11-13 13:20 H22 hypothesis by AC1: When methods are compared over many series or data sets, the distribution of their scaled errors across the series should be reported, not only the mean, becau… [a_priori] 58 2026-11-13 13:20 H23 hypothesis by AC1: When several methods are compared over many data sets, their differences should be tested with the [[TM25:Friedman test]] on ranks followed by [[TM26:Nemenyi t… [a_priori] 59 2026-11-13 13:20 H24 hypothesis by AC1: When the question is which methods differ from the best one over many series, multiple comparisons with the best ([[TM27:MCB]]) on ranks answer it, as in the s… [a_priori] 60 2026-11-13 13:20 H25 hypothesis by AC1: The accuracy of two forecasts of the same series should be compared with the [[TM28:Diebold-Mariano test]], which allows forecast errors that are serially corr… [a_priori] 61 2026-11-13 13:20 H26 hypothesis by AC1: An error measure should not be chosen because a competition or a well-known study used it: the measure has to be consistent for what the compared methods predi… [a_priori] 62 2026-11-13 13:24 SR1 source by AC1: Making and evaluating point forecasts (arXiv:0912.0902, the preprint of Gneiting 2011, JASA 106(494), 746-762, doi:10.1198/jasa.2011.r10138) · arXiv · private 63 2026-11-13 13:24 SR2 source by AC1: Strictly proper scoring rules, prediction, and estimation (Gneiting and Raftery 2007, JASA 102(477), 359-378, doi:10.1198/016214506000001437), author's copy ·… 64 2026-11-13 13:24 SR3 source by AC1: Probabilistic forecasts, calibration and sharpness (Gneiting, Balabdaoui and Raftery 2007, JRSS B 69(2), 243-268, doi:10.1111/j.1467-9868.2007.00587.x), author… 65 2026-11-13 13:24 SR4 source by AC1: Statistical comparisons of classifiers over multiple data sets (Demsar 2006, JMLR 7, 1-30) · Journal of Machine Learning Research · private 66 2026-11-13 13:25 SR5 source by AC1: Automatic time series forecasting: the forecast package for R (Hyndman and Khandakar 2008, JSS 27(3), doi:10.18637/jss.v027.i03) · Journal of Statistical Softw… 67 2026-11-13 13:25 SR6 source by AC1: Forecasting: principles and practice, 2nd edition, section 3.2 Transformations and adjustments (Hyndman and Athanasopoulos 2018) · OTexts · private 68 2026-11-13 13:25 SR7 source by AC1: Another look at measures of forecast accuracy (preprint of Hyndman and Koehler 2006, IJF 22(4), 679-688, doi:10.1016/j.ijforecast.2006.03.001) · International… 69 2026-11-13 13:25 SR8 source by AC1: The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets (Saito and Rehmsmeier 2015, PLoS ONE… 70 2026-11-13 13:28 NT2 note by AC1: Kolassa, S. (2020). Why the 'best' point forecast depends on the error or accuracy measure. International Journal of Forecasting 36(1), 208-211, doi:10.1016/j.… 71 2026-11-13 13:28 NT3 note by AC1: Makridakis, S., Spiliotis, E., and Assimakopoulos, V. (2022). M5 accuracy competition: results, findings, and conclusions. International Journal of Forecasting… 72 2026-11-13 13:28 NT4 note by AC1: Makridakis, S., Spiliotis, E., Assimakopoulos, V., Chen, Z., Gaba, A., Tsetlin, I., and Winkler, R. L. (2022). The M5 uncertainty competition: results, finding… 73 2026-11-13 13:28 NT5 note by AC1: Goodwin, P., and Lawton, R. (1999). On the asymmetry of the symmetric MAPE. International Journal of Forecasting 15(4), 405-408, doi:10.1016/S0169-2070(99)0000… 74 2026-11-13 13:28 NT6 note by AC1: Kolassa, S. (2016). Evaluating predictive count data distributions in retail sales forecasting. International Journal of Forecasting 32(3), 788-803, doi:10.101… 75 2026-11-13 13:29 NT7 note by AC1: Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review 78(1), 1-3, doi:10.1175/1520-0493(1950)078<0001:VOFEIT… 76 2026-11-13 13:29 NT8 note by AC1: Hand, D. J. (2009). Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine Learning 77(1), 103-123, doi:10.1007/s109… 77 2026-11-13 13:29 NT9 note by AC1: Koning, A. J., Franses, P. H., Hibon, M., and Stekler, H. O. (2005). The M3 competition: statistical tests of the results. International Journal of Forecasting… 78 2026-11-13 13:29 NT10 note by AC1: Diebold, F. X., and Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business and Economic Statistics 13(3), 253-263, doi:10.1080/07350015.1995… 79 2026-11-13 13:29 NT11 note by AC1: No paper among those the guide cites is the source for avoiding accuracy as the measure of class probabilities. The claim rests on an argument from the definit… 80 2026-11-13 13:30 ST1 status by AC1: H1 -> supported, evidence SR1, NT2: Gneiting (2011, abstract, SR1): 'If the forecaster receives a directive in the form of a functional, it is critical tha… 81 2026-11-13 13:30 ST2 status by AC1: H2 -> supported, evidence SR1: Gneiting (2011, SR1): 'the squared error scoring function is consistent for the mean' (Section 5, Discussion); the scor… 82 2026-11-13 13:30 ST3 status by AC1: H3 -> supported, evidence NT3, SR1: The M5 accuracy competition (NT3) ranked point forecasts by the RMSSE: the RMSE of each series scaled by the in-sample… 83 2026-11-13 13:30 ST4 status by AC1: H4 -> supported, evidence SR5: Hyndman and Khandakar (2008, SR5, Section 2.2): for many models the conditional means are identical to the point foreca… 84 2026-11-13 13:30 ST5 status by AC1: H5 -> supported, evidence SR6: Hyndman and Athanasopoulos (2018, SR6, Section 3.2): 'the back-transformed point forecast will not be the mean of the f… 85 2026-11-13 13:30 ST6 status by AC1: H6 -> supported, evidence SR1: Gneiting (2011, SR1, Section 5): squared error is consistent for the mean, 'and the absolute error scoring function for… 86 2026-11-13 13:30 ST7 status by AC1: H7 -> supported, evidence SR1: Gneiting (2011, SR1, Section 5): the absolute percentage error 'is consistent for a non-standard functional, namely, th… 87 2026-11-13 13:30 ST8 status by AC1: H8 -> supported, evidence SR7, NT5: Hyndman and Koehler (2006, SR7, Section 2): the 'symmetric' measures 'are not as symmetric as their name suggests': for… 88 2026-11-13 13:30 ST9 status by AC1: H9 -> supported, evidence SR1, SR7: Gneiting (2011, SR1, Section 5): 'the absolute error scoring function' is consistent 'for the median'. Hyndman and Koeh… 89 2026-11-13 13:30 ST10 status by AC1: H10 -> supported, evidence NT6, NT3: Kolassa (2016, NT6): on intermittent count data, absolute-error measures are minimised by a forecast of zero and are un… 90 2026-11-13 13:30 ST11 status by AC1: H11 -> supported, evidence SR1, NT4: Gneiting (2011, SR1, abstract and Theorem 3.3): a scoring function 'is consistent for a quantile if and only if it is g… 91 2026-11-13 13:31 ST12 status by AC1: H12 -> supported, evidence SR2: Gneiting and Raftery (2007, SR2, Section 1 and Section 6.2): they 'discuss the interval score, a scoring rule for predi… 92 2026-11-13 13:31 ST13 status by AC1: H13 -> supported, evidence SR2: Gneiting and Raftery (2007, SR2, abstract) propose the interval score because it 'addresses width as well as coverage':… 93 2026-11-13 13:31 ST14 status by AC1: H14 -> supported, evidence SR2: Gneiting and Raftery (2007, SR2): 'The CRPS is proper relative to the class P' (Section 4.2) and 'The logarithmic score… 94 2026-11-13 13:31 ST15 status by AC1: H15 -> supported, evidence SR3: Gneiting, Balabdaoui and Raftery (2007, SR3, abstract): evaluation 'based on the paradigm of maximizing the sharpness o… 95 2026-11-13 13:31 ST16 status by AC1: H16 -> supported, evidence NT6: Kolassa (2016, NT6): predictive distributions of count data are evaluated with proper scoring rules and a randomised pr… 96 2026-11-13 13:31 ST17 status by AC1: H17 -> supported, evidence SR2, NT7: Gneiting and Raftery (2007, SR2, Section 3, Example 1, 'Quadratic or Brier score', and the logarithmic score): both are… 97 2026-11-13 13:31 ST18 status by AC1: H18 -> supported, evidence NT11: The argument of NT11: accuracy uses a probability only through which side of a threshold it falls and counts every erro… 98 2026-11-13 13:31 ST19 status by AC1: H19 -> supported, evidence NT8: Hand (2009, NT8): the AUC averages the misclassification loss over a cost distribution that depends on the classifier,… 99 2026-11-13 13:31 ST20 status by AC1: H20 -> supported, evidence SR8: Saito and Rehmsmeier (2015, SR8, title and abstract): 'The Precision-Recall Plot Is More Informative than the ROC Plot… 100 2026-11-13 13:31 ST21 status by AC1: H21 -> supported, evidence SR1, NT2: Kolassa (2020, NT2): different error measures are minimised by different summaries of the same predictive density, so n… 101 2026-11-13 13:31 ST22 status by AC1: H22 -> supported, evidence SR4: Demsar (2006, SR4, Section 3.1): 'Averages are also susceptible to outliers. They allow classifier's excellent performa… 102 2026-11-13 13:31 ST23 status by AC1: H23 -> supported, evidence SR4: Demsar (2006, SR4, abstract): 'we recommend a set of simple, yet safe and robust non-parametric tests', 'the Friedman t… 103 2026-11-13 13:31 ST24 status by AC1: H24 -> supported, evidence NT9: Koning et al. (2005, NT9) test the M3 results with non-parametric multiple comparisons on ranks over series, including… 104 2026-11-13 13:31 ST25 status by AC1: H25 -> supported, evidence NT10: Diebold and Mariano (1995, NT10) propose tests of no difference in the accuracy of two competing forecasts that allow n… 105 2026-11-13 13:31 ST26 status by AC1: H26 -> supported, evidence SR1, NT2: Kolassa (2020, NT2), a commentary on the measures of the M4 competition: the best point forecast depends on the error m… 106 2026-11-13 13:32 ST27 status by AC1: H7 -> supported, evidence SR1: Gneiting (2011, SR1, Section 5): the absolute percentage error 'is consistent for a non-standard functional, namely, th… 107 2026-11-13 13:44 DC1 decision by AC1: The supervisor left the implementation facts out of the study: 'Implementation facts are not needed here. This is a side thing, which belongs to ETS, not this… 108 2026-11-13 13:44 ST28 status by AC1: H4 -> abandoned: Out of the study's scope by the supervisor's decision DC1: a property of particular models, not a rule for evaluating p… 109 2026-11-13 13:44 ST29 status by AC1: H5 -> abandoned: Out of the study's scope by the supervisor's decision DC1: a property of particular models, not a rule for evaluating p… 110 2026-11-13 13:45 H27 hypothesis by AC1: [[TM10:sMAPE]] should not be the measure for point forecasts: for the same outcome it penalises forecasts below the outcome more heavily than forecasts above i… [a_priori] 111 2026-11-13 13:45 H28 hypothesis by AC1: Scale-dependent error measures ([[TM5:RMSE]], [[TM7:MAE]]) should not be summed or averaged across series on different scales: the series with the largest valu… [a_priori] 112 2026-11-13 13:45 H29 hypothesis by AC1: To aggregate the accuracy of point forecasts over series on different scales, each series' errors should first be scaled by a quantity of that series (as in [[… [a_priori] 113 2026-11-13 13:51 NT12 note by AC1: Athanasopoulos, G., and Kourentzes, N. (2023). On the evaluation of hierarchical forecasts. International Journal of Forecasting 39(4), 1502-1511, doi:10.1016/… 114 2026-11-13 13:51 ST30 status by AC1: H27 -> supported, evidence SR7, NT5: Hyndman and Koehler (2006, SR7, Section 2.2): for the same outcome, sMAPE 'has a heavier penalty when forecasts are low… 115 2026-11-13 13:51 ST31 status by AC1: H28 -> supported, evidence SR7, NT12: Hyndman and Koehler (2006, SR7, Section 2.1): scale-dependent measures 'are useful when comparing different methods on… 116 2026-11-13 13:51 ST32 status by AC1: H29 -> supported, evidence SR7, NT3: Hyndman and Koehler (2006, SR7, Section 3): 'measures based on scaled errors should become the standard approach in com… 117 2026-11-13 13:51 ST33 status by AC1: H28 -> supported, evidence SR7, NT12: Hyndman and Koehler (2006, SR7, Section 2.1): scale-dependent measures 'are useful when comparing different methods on… 118 2026-11-13 14:05 SR9 source by AC1: The M5 Competition: Competitors' Guide (the organisers' definition of the RMSSE, the scaled pinball loss and the reasons for them) · M Open Forecasting Center,… 119 2026-11-13 14:06 NT13 note by AC1: Makridakis, S., Spiliotis, E., and Assimakopoulos, V. (2020). The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Fore… 120 2026-11-13 14:07 H30 hypothesis by AC1: Quantile forecasts should be evaluated with the [[TM12:pinball loss]] at the quantile's level, the scoring function consistent for that quantile, averaged over… [a_priori] 121 2026-11-13 14:07 H31 hypothesis by AC1: When methods are compared over many series, their accuracy should be summarised by aggregates of a scaled measure across the series, the mean together with the… [a_priori] 122 2026-11-13 14:07 H32 hypothesis by AC1: To tell whether several methods whose aggregate accuracy is similar differ significantly over many series or data sets, their ranks on each series should be te… [a_priori] 123 2026-11-13 14:07 H33 hypothesis by AC1: To tell which methods differ significantly from the best one over many series, multiple comparisons with the best ([[TM27:MCB]]) on ranks should be used, as in… [a_priori] 124 2026-11-13 14:07 H34 hypothesis by AC1: When quantile forecasts are compared over series on different scales, each series' [[TM12:pinball loss]] should be scaled by a quantity of that series, as the… [a_priori] 125 2026-11-13 14:08 H35 hypothesis by AC1: When prediction intervals are compared over series on different scales, each series' [[TM13:interval score]] should be scaled by a quantity of that series, as… [a_priori] 126 2026-11-13 14:08 ST34 status by AC1: H3 -> supported, evidence SR9, NT3, SR1: The M5 Competitors' Guide (SR9, Point forecasts): the RMSSE divides each series' squared errors by the in-sample mean s… 127 2026-11-13 14:08 ST35 status by AC1: H10 -> supported, evidence SR9, NT6: The M5 Competitors' Guide (SR9, Point forecasts): on intermittent series 'absolute errors, which are optimized for the… 128 2026-11-13 14:08 ST36 status by AC1: H30 -> supported, evidence SR1: Gneiting (2011, SR1, abstract and Theorem 3.3): a scoring function 'is consistent for a quantile if and only if it is g… 129 2026-11-13 14:08 ST37 status by AC1: H34 -> supported, evidence SR9, NT4: The M5 Competitors' Guide (SR9, Probabilistic forecasts): forecasts of quantiles are evaluated with the 'Scaled Pinball… 130 2026-11-13 14:08 ST38 status by AC1: H35 -> supported, evidence NT13, SR7, SR2: The M4 competition (NT13) evaluated prediction intervals with the mean scaled interval score, the interval score of Gne… 131 2026-11-13 14:08 ST39 status by AC1: H31 -> supported, evidence SR4, SR7, SR9: Demsar (2006, SR4, Section 3.1): 'Averages are also susceptible to outliers'. Hyndman and Koehler (2006, SR7, Section 3… 132 2026-11-13 14:08 ST40 status by AC1: H32 -> supported, evidence SR4: Demsar (2006, SR4, abstract; Sections 3.1 and 3.2.2): it recommends 'the Friedman test with the corresponding post-hoc… 133 2026-11-13 14:08 ST41 status by AC1: H33 -> supported, evidence NT9: Koning et al. (2005, NT9) test the M3 results with multiple comparisons with the best (MCB) on ranks over series, which… 134 2026-11-13 14:08 ST42 status by AC1: H29 -> supported, evidence SR7, SR9, NT3: Hyndman and Koehler (2006, SR7, Section 3): 'measures based on scaled errors should become the standard approach in com… 135 2026-11-13 16:14 NT14 note by AC1: Ivan Svetunkov (AC2) reported in this session on 2026-11-13 that he read in full, and checked against what this study cites them for: Kolassa (2016); Kolassa (… 136 2026-11-13 16:14 EX1 extension by AC1: review 137 2026-11-13 16:14 SC1 screening by AC1: doi:10.1198/jasa.2011.r10138 full_text: include, read full_text 138 2026-11-13 16:14 SC2 screening by AC1: doi:10.1198/016214506000001437 full_text: include, read full_text 139 2026-11-13 16:14 SC3 screening by AC1: doi:10.1111/j.1467-9868.2007.00587.x full_text: include, read full_text 140 2026-11-13 16:14 SC4 screening by AC1: url:https://jmlr.org/papers/v7/demsar06a.html full_text: include, read full_text 141 2026-11-13 16:14 SC5 screening by AC1: doi:10.18637/jss.v027.i03 full_text: include, read full_text 142 2026-11-13 16:14 SC6 screening by AC1: url:https://otexts.com/fpp2/transformations.html full_text: include, read full_text 143 2026-11-13 16:14 SC7 screening by AC1: doi:10.1016/j.ijforecast.2006.03.001 full_text: include, read full_text 144 2026-11-13 16:14 SC8 screening by AC1: doi:10.1371/journal.pone.0118432 full_text: include, read full_text 145 2026-11-13 16:14 SC9 screening by AC1: url:https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf full_text: include, read full_text 146 2026-11-13 16:14 SC10 screening by AC1: doi:10.1016/j.ijforecast.2019.02.017 full_text: include, read full_text 147 2026-11-13 16:14 SC11 screening by AC1: doi:10.1016/j.ijforecast.2021.11.013 full_text: include, read full_text 148 2026-11-13 16:15 SC12 screening by AC1: doi:10.1016/j.ijforecast.2021.10.009 full_text: include, read full_text 149 2026-11-13 16:15 SC13 screening by AC1: doi:10.1016/S0169-2070(99)00007-2 full_text: include, read full_text 150 2026-11-13 16:15 SC14 screening by AC1: doi:10.1016/j.ijforecast.2015.12.004 full_text: include, read full_text 151 2026-11-13 16:15 SC15 screening by AC1: doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 full_text: include, read abstract 152 2026-11-13 16:15 SC16 screening by AC1: doi:10.1007/s10994-009-5119-5 full_text: include, read abstract 153 2026-11-13 16:15 SC17 screening by AC1: doi:10.1016/j.ijforecast.2004.10.003 full_text: include, read full_text 154 2026-11-13 16:15 SC18 screening by AC1: doi:10.1080/07350015.1995.10524599 full_text: include, read full_text 155 2026-11-13 16:15 SC19 screening by AC1: doi:10.1016/j.ijforecast.2022.08.003 full_text: include, read full_text 156 2026-11-13 16:15 SC20 screening by AC1: doi:10.1016/j.ijforecast.2019.04.014 full_text: include, read full_text 157 2026-11-13 16:16 H36 hypothesis by AC1: Squared error is a [[TM1:consistent scoring function]] for the mean. [a_priori] 158 2026-11-13 16:16 XT1 extract by AC1: doi:10.1198/jasa.2011.r10138 Section 5 (Discussion); Theorem 3.1 (arXiv:0912.0902 numbering) for H36: "the squared error scoring function is consistent for the… 159 2026-11-13 16:16 ST43 status by AC1: H36 -> supported, evidence XT1, SR1: Gneiting (2011), Section 5 and Theorem 3.1. 160 2026-11-13 16:16 H37 hypothesis by AC1: Which point forecast has the lowest expected score depends on the scoring function: a [[TM1:consistent scoring function]] for a functional, such as the mean or… [a_priori] 161 2026-11-13 16:16 XT2 extract by AC1: doi:10.1198/jasa.2011.r10138 Abstract (arXiv:0912.0902 numbering) for H37: "it is critical that the scoring function be consistent for it" 162 2026-11-13 16:16 ST44 status by AC1: H37 -> supported, evidence XT2, SR1, NT2, NT14: Gneiting (2011, abstract) defines consistency and states that the scoring function must be consistent for the functiona… 163 2026-11-13 16:16 H38 hypothesis by AC1: Absolute error is a [[TM1:consistent scoring function]] for the median. [a_priori] 164 2026-11-13 16:16 XT3 extract by AC1: doi:10.1198/jasa.2011.r10138 Section 5 (Discussion) (arXiv:0912.0902 numbering) for H38: "the absolute error scoring function for the median" 165 2026-11-13 16:16 ST45 status by AC1: H38 -> supported, evidence XT3, SR1: Gneiting (2011), Section 5. 166 2026-11-13 16:16 H39 hypothesis by AC1: The absolute percentage error averaged by [[TM9:MAPE]] is consistent for the [[TM11:median of order -1]], a functional that supports severe underforecasts comp… [a_priori] 167 2026-11-13 16:16 XT4 extract by AC1: doi:10.1198/jasa.2011.r10138 Section 5 (Discussion); the beta-median in Section 1.3 (arXiv:0912.0902 numbering) for H39: "the median of order −1" 168 2026-11-13 16:17 XT5 extract by AC1: doi:10.1198/jasa.2011.r10138 Section 5 (Discussion) (arXiv:0912.0902 numbering) for H39: "severe underforecasts, as compared to the mean or median" 169 2026-11-13 16:17 ST46 status by AC1: H39 -> supported, evidence XT4, XT5, SR1: Gneiting (2011), Section 5. 170 2026-11-13 16:17 H40 hypothesis by AC1: For the same outcome, [[TM10:sMAPE]] penalises a forecast below the outcome more heavily than a forecast above it. [a_priori] 171 2026-11-13 16:17 XT6 extract by AC1: doi:10.1016/j.ijforecast.2006.03.001 Section 2.2 (Measures based on percentage errors) for H40: "has a heavier penalty when forecasts are low compared to when… 172 2026-11-13 16:17 ST47 status by AC1: H40 -> supported, evidence XT6, SR7, NT5, NT14: Hyndman and Koehler (2006), Section 2.2; Goodwin and Lawton (1999), read in full by the guarantor (NT14). 173 2026-11-13 16:17 H41 hypothesis by AC1: The [[TM12:pinball loss]] at a level is a [[TM1:consistent scoring function]] for the quantile at that level. [a_priori] 174 2026-11-13 16:17 XT7 extract by AC1: doi:10.1198/jasa.2011.r10138 Abstract; Theorem 3.3 (arXiv:0912.0902 numbering) for H41: "It is consistent for a quantile if and only if it is generalized piece… 175 2026-11-13 16:17 ST48 status by AC1: H41 -> supported, evidence XT7, SR1: Gneiting (2011), abstract and Theorem 3.3: the scoring functions consistent for a quantile are the generalized piecewis… 176 2026-11-13 16:17 H42 hypothesis by AC1: The [[TM13:interval score]] is a [[TM2:proper scoring rule]] for central prediction intervals that addresses their width as well as their coverage. [a_priori] 177 2026-11-13 16:17 XT8 extract by AC1: doi:10.1198/016214506000001437 Section 1 (Introduction) for H42: "a scoring rule for prediction intervals that is proper" 178 2026-11-13 16:17 XT9 extract by AC1: doi:10.1198/016214506000001437 Abstract for H42: "propose the intuitively appealing interval score as a utility function in interval estimation that addresses… 179 2026-11-13 16:17 ST49 status by AC1: H42 -> supported, evidence XT8, XT9, SR2: Gneiting and Raftery (2007), abstract, Section 1 and Section 6.2. 180 2026-11-13 16:17 H43 hypothesis by AC1: The [[TM15:CRPS]] is a [[TM2:proper scoring rule]] for predictive distributions of continuous outcomes. [a_priori] 181 2026-11-13 16:17 XT10 extract by AC1: doi:10.1198/016214506000001437 Section 4.2 (Continuous ranked probability score) for H43: "The CRPS is proper relative to the class P" 182 2026-11-13 16:17 ST50 status by AC1: H43 -> supported, evidence XT10, SR2: Gneiting and Raftery (2007), Section 4.2. 183 2026-11-13 16:17 H44 hypothesis by AC1: The [[TM16:log score]] is a strictly [[TM2:proper scoring rule]]. [a_priori] 184 2026-11-13 16:17 XT11 extract by AC1: doi:10.1198/016214506000001437 Section 4.1 for H44: "The logarithmic score is strictly proper" 185 2026-11-13 16:17 ST51 status by AC1: H44 -> supported, evidence XT11, SR2: Gneiting and Raftery (2007), Section 4.1. 186 2026-11-13 16:17 H45 hypothesis by AC1: The [[TM17:Brier score]] is a [[TM2:proper scoring rule]] for forecast probabilities of classes. [a_priori] 187 2026-11-13 16:17 XT12 extract by AC1: doi:10.1198/016214506000001437 Section 3, Example 1 (Quadratic or Brier score) for H45: "Example 1 (Quadratic or Brier score)" 188 2026-11-13 16:17 ST52 status by AC1: H45 -> supported, evidence XT12, SR2, NT7: Gneiting and Raftery (2007), Section 3, Example 1; Brier (1950), checked by its abstract and bibliographic record only… 189 2026-11-13 16:17 H46 hypothesis by AC1: Predictive distributions are assessed by maximising their [[TM20:sharpness]] subject to their [[TM19:calibration]]. [a_priori] 190 2026-11-13 16:18 XT13 extract by AC1: doi:10.1111/j.1467-9868.2007.00587.x Abstract for H46: "is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibra… 191 2026-11-13 16:18 ST53 status by AC1: H46 -> supported, evidence XT13, SR3: Gneiting, Balabdaoui and Raftery (2007), abstract and Sections 3.1 and 3.3. 192 2026-11-13 16:18 H47 hypothesis by AC1: The calibration of predictive distributions of counts can be assessed with a randomised [[TM18:probability integral transform]]. [a_priori] 193 2026-11-13 16:18 ST54 status by AC1: H47 -> supported, evidence NT6, NT14: Kolassa (2016), read in full by the guarantor (NT14). 194 2026-11-13 16:18 H48 hypothesis by AC1: On [[TM24:intermittent series]], absolute-error measures give their best scores to forecasts close to zero. [a_priori] 195 2026-11-13 16:18 XT14 extract by AC1: url:https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf Point forecasts for H48: "absolute errors, which are optimized for the med… 196 2026-11-13 16:18 ST55 status by AC1: H48 -> supported, evidence XT14, SR9, NT6, NT14: The M5 competitors' guide (Point forecasts); Kolassa (2016), read in full by the guarantor (NT14). 197 2026-11-13 16:18 H49 hypothesis by AC1: The [[TM22:AUC]] is equivalent to averaging the misclassification loss over a distribution of misclassification costs that depends on the classifier. [a_priori] 198 2026-11-13 16:18 ST56 status by AC1: H49 -> supported, evidence NT8: Hand (2009), checked by its abstract only (NT8). 199 2026-11-13 16:18 H50 hypothesis by AC1: On imbalanced data, the [[TM23:precision-recall curve]] is more informative than the ROC curve for evaluating binary classifiers. [a_priori] 200 2026-11-13 16:18 XT15 extract by AC1: doi:10.1371/journal.pone.0118432 Title for H50: "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanc… 201 2026-11-13 16:18 XT16 extract by AC1: doi:10.1371/journal.pone.0118432 Abstract for H50: "can be deceptive" 202 2026-11-13 16:18 ST57 status by AC1: H50 -> supported, evidence XT15, XT16, SR8: Saito and Rehmsmeier (2015), title and abstract. 203 2026-11-13 16:18 H51 hypothesis by AC1: Scale-dependent error measures, such as [[TM5:RMSE]] and [[TM7:MAE]], cannot be compared across series on different scales. [a_priori] 204 2026-11-13 16:18 XT17 extract by AC1: doi:10.1016/j.ijforecast.2006.03.001 Section 2.1 (Scale-dependent measures) for H51: "should not be used, for example, when comparing across data sets that hav… 205 2026-11-13 16:18 XT18 extract by AC1: doi:10.1016/j.ijforecast.2006.03.001 Section 2.4 (Relative measures) for H51: "It makes no sense to compute the MAE across series (due to their different scale… 206 2026-11-13 16:18 ST58 status by AC1: H51 -> supported, evidence XT17, XT18, SR7: Hyndman and Koehler (2006), Sections 2.1 and 2.4. 207 2026-11-13 16:18 H52 hypothesis by AC1: When the errors of series on different scales are summed or averaged without scaling, the series with the largest values dominate the result. [a_priori] 208 2026-11-13 16:18 ST59 status by AC1: H52 -> supported, evidence NT12, NT14: Athanasopoulos and Kourentzes (2023), read in full by the guarantor (NT14). 209 2026-11-13 16:18 H53 hypothesis by AC1: Scaled errors, as in [[TM8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. [a_priori] 210 2026-11-13 16:18 XT19 extract by AC1: doi:10.1016/j.ijforecast.2006.03.001 Abstract; Section 3 (Scaled errors) for H53: "we propose that the mean absolute scaled error become the standard measure f… 211 2026-11-13 16:18 ST60 status by AC1: H53 -> supported, evidence XT19, SR7: Hyndman and Koehler (2006), abstract and Section 3. 212 2026-11-13 16:18 H54 hypothesis by AC1: The M5 accuracy competition scored point forecasts with the [[TM6:RMSSE]]: each series' squared errors divided by the in-sample mean squared error of its one-s… [a_priori] 213 2026-11-13 16:18 XT20 extract by AC1: url:https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf Point forecasts for H54: "The accuracy of the point forecasts will be eval… 214 2026-11-13 16:18 XT21 extract by AC1: url:https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf Point forecasts for H54: "The measure is scale independent, meaning that i… 215 2026-11-13 16:18 ST61 status by AC1: H54 -> supported, evidence XT20, XT21, SR9, NT3, NT14: The M5 competitors' guide (Point forecasts); the M5 accuracy paper, read in full by the guarantor (NT14). 216 2026-11-13 16:18 H55 hypothesis by AC1: The M5 uncertainty competition scored quantile forecasts with the scaled pinball loss: each series' [[TM12:pinball loss]] divided by the in-sample mean absolut… [a_priori] 217 2026-11-13 16:18 XT22 extract by AC1: url:https://github.com/Mcompetitions/M5-methods/blob/master/M5-Competitors-Guide.pdf Probabilistic forecasts for H55: "The precision of the probabilistic forec… 218 2026-11-13 16:18 ST62 status by AC1: H55 -> supported, evidence XT22, SR9, NT4, NT14: The M5 competitors' guide (Probabilistic forecasts); the M5 uncertainty paper, read in full by the guarantor (NT14). 219 2026-11-13 16:19 H56 hypothesis by AC1: The M4 competition scored prediction intervals with the mean scaled interval score: the [[TM13:interval score]] of each series scaled by an in-sample error of… [a_priori] 220 2026-11-13 16:19 ST63 status by AC1: H56 -> supported, evidence NT13, NT14: The M4 paper, read in full by the guarantor (NT14). 221 2026-11-13 16:19 H57 hypothesis by AC1: Averages of a performance measure over many data sets are susceptible to outliers. [a_priori] 222 2026-11-13 16:19 XT23 extract by AC1: url:https://jmlr.org/papers/v7/demsar06a.html Section 3.1 (Comparisons of two classifiers) for H57: "Averages are also susceptible to outliers" 223 2026-11-13 16:19 ST64 status by AC1: H57 -> supported, evidence XT23, SR4: Demsar (2006), Section 3.1. 224 2026-11-13 16:19 H58 hypothesis by AC1: The [[TM25:Friedman test]], followed by the [[TM26:Nemenyi test]], compares several methods over many data sets by their ranks on each data set, assuming only… [a_priori] 225 2026-11-13 16:19 XT24 extract by AC1: url:https://jmlr.org/papers/v7/demsar06a.html Section 3.2.2 (Friedman test) for H58: "It ranks the algorithms for each data set separately" 226 2026-11-13 16:19 XT25 extract by AC1: url:https://jmlr.org/papers/v7/demsar06a.html Section 3.2.2 (Friedman test) for H58: "The Nemenyi test" 227 2026-11-13 16:19 XT26 extract by AC1: url:https://jmlr.org/papers/v7/demsar06a.html Section 5 (Conclusion) for H58: "They are appropriate since they assume some, but limited commensurability" 228 2026-11-13 16:19 ST65 status by AC1: H58 -> supported, evidence XT24, XT25, XT26, SR4: Demsar (2006), Sections 3.2.2 and 5. 229 2026-11-13 16:19 H59 hypothesis by AC1: [[TM27:MCB]] on ranks was used to test the differences between the methods of the M3 competition. [a_priori] 230 2026-11-13 16:19 ST66 status by AC1: H59 -> supported, evidence NT9, NT14: Koning, Franses, Hibon and Stekler (2005), read in full by the guarantor (NT14). 231 2026-11-13 16:19 H60 hypothesis by AC1: The [[TM28:Diebold-Mariano test]] of equal accuracy of two forecasts allows forecast errors that are serially correlated and not normally distributed. [a_priori] 232 2026-11-13 16:19 ST67 status by AC1: H60 -> supported, evidence NT10, NT14: Diebold and Mariano (1995), read in full by the guarantor (NT14). 233 2026-11-13 16:20 H61 hypothesis by AC1: [[TM10:sMAPE]] should not be the measure for point forecasts. [a_priori] 234 2026-11-13 16:20 ST68 status by AC1: H61 -> supported, evidence H40: sMAPE penalises forecasts below the outcome more heavily than forecasts above it, so it rewards forecasts that are too… 235 2026-11-13 16:20 H62 hypothesis by AC1: Point forecasts should be evaluated with a [[TM1:consistent scoring function]] for the functional they target: the mean, the median or a quantile. [a_priori] 236 2026-11-13 16:20 ST69 status by AC1: H62 -> supported, evidence H37: A forecast is built to target one functional, and only a scoring function consistent for that functional is lowest in e… 237 2026-11-13 16:20 H63 hypothesis by AC1: [[TM3:Point forecasts of the mean]] should be evaluated with squared error, as in [[TM5:RMSE]]. [a_priori] 238 2026-11-13 16:20 ST70 status by AC1: H63 -> supported, evidence H36, H37: Squared error is consistent for the mean, so its expected value is lowest for the forecast that equals the mean; the RM… 239 2026-11-13 16:20 H64 hypothesis by AC1: When [[TM3:point forecasts of the mean]] are compared over series on different scales, each series' [[TM5:RMSE]] should be scaled by the in-sample RMSE of its… [a_priori] 240 2026-11-13 16:20 ST71 status by AC1: H64 -> supported, evidence H54, H36, H53: Dividing a series' squared errors by a fixed positive number, the in-sample mean squared error of its one-step naive fo… 241 2026-11-13 16:20 H65 hypothesis by AC1: [[TM7:MAE]] and [[TM8:MASE]] should not be the main measure for [[TM3:point forecasts of the mean]]. [a_priori] 242 2026-11-13 16:20 ST72 status by AC1: H65 -> supported, evidence H38, H36: Absolute error, which MAE averages and MASE scales, is consistent for the median, not the mean; on skewed outcomes the… 243 2026-11-13 16:20 H66 hypothesis by AC1: [[TM9:MAPE]] should not be the measure for point forecasts of the mean or the median. [a_priori] 244 2026-11-13 16:20 ST73 status by AC1: H66 -> supported, evidence H39: MAPE is consistent for the median of order -1, which supports severe underforecasts compared with the mean or the media… 245 2026-11-13 16:20 H67 hypothesis by AC1: [[TM4:Point forecasts of the median]] should be evaluated with absolute error, as in [[TM7:MAE]] or, across series on different scales, [[TM8:MASE]]. [a_priori] 246 2026-11-13 16:20 ST74 status by AC1: H67 -> supported, evidence H38, H53: Absolute error is consistent for the median, so its expected value is lowest for the median forecast; MASE divides each… 247 2026-11-13 16:20 H68 hypothesis by AC1: On [[TM24:intermittent series]], absolute-error measures should not be used alone. [a_priori] 248 2026-11-13 16:21 ST75 status by AC1: H68 -> supported, evidence H48, H38: On intermittent series the median of the outcome can be zero, and absolute error, consistent for the median, then gives… 249 2026-11-13 16:21 H69 hypothesis by AC1: Prediction intervals should be evaluated with the [[TM13:interval score]], reported together with their [[TM14:coverage]] against the nominal level. [a_priori] 250 2026-11-13 16:21 ST76 status by AC1: H69 -> supported, evidence H42: The interval score is proper for central intervals and addresses width as well as coverage; coverage against the nomina… 251 2026-11-13 16:21 H70 hypothesis by AC1: [[TM14:Coverage]] alone should not be the measure of prediction intervals. [a_priori] 252 2026-11-13 16:21 ST77 status by AC1: H70 -> supported, evidence H42: Coverage counts only whether each outcome falls inside its interval; widening an interval never lowers its coverage, so… 253 2026-11-13 16:21 H71 hypothesis by AC1: Predictive distributions should be evaluated with a [[TM2:proper scoring rule]], such as the [[TM15:CRPS]] or the [[TM16:log score]]. [a_priori] 254 2026-11-13 16:21 ST78 status by AC1: H71 -> supported, evidence H43, H44: The CRPS and the log score are proper: their expected values are best for the true distribution, so they never reward a… 255 2026-11-13 16:21 H72 hypothesis by AC1: The evaluation of predictive distributions should check their [[TM19:calibration]] and report their [[TM20:sharpness]]. [a_priori] 256 2026-11-13 16:21 ST79 status by AC1: H72 -> supported, evidence H46: Predictive distributions are assessed by maximising their sharpness subject to calibration; checking calibration, with… 257 2026-11-13 16:21 H73 hypothesis by AC1: Predictive distributions of counts, such as forecasts of intermittent sales, should be evaluated with proper scoring rules and a randomised [[TM18:probability… [a_priori] 258 2026-11-13 16:21 ST80 status by AC1: H73 -> supported, evidence H47, H43, H44: The probability integral transform of a discrete distribution takes few values and is not uniform even when the forecas… 259 2026-11-13 16:21 H74 hypothesis by AC1: Forecast probabilities of classes should be evaluated with a [[TM2:proper scoring rule]], such as the [[TM16:log score]] or the [[TM17:Brier score]]. [a_priori] 260 2026-11-13 16:21 ST81 status by AC1: H74 -> supported, evidence H45, H44: The Brier score is proper and the log score strictly proper, so both are best in expectation for the true probabilities… 261 2026-11-13 16:21 H75 hypothesis by AC1: [[TM21:Accuracy]] should not be the measure of forecast class probabilities. [a_priori] 262 2026-11-13 16:21 ST82 status by AC1: H75 -> supported, evidence H45, H44: Accuracy uses a forecast probability only through which side of a threshold it falls, and counts every misclassificatio… 263 2026-11-13 16:21 H76 hypothesis by AC1: [[TM22:AUC]] should not be the measure of forecast class probabilities. [a_priori] 264 2026-11-13 16:21 ST83 status by AC1: H76 -> supported, evidence H49: The AUC averages the misclassification loss over a cost distribution that depends on the classifier, so different class… 265 2026-11-13 16:21 H77 hypothesis by AC1: With rare positive cases, the [[TM23:precision-recall curve]] should be reported rather than the ROC curve alone. [a_priori] 266 2026-11-13 16:21 ST84 status by AC1: H77 -> supported, evidence H50: On imbalanced data the ROC curve can be deceptive about how reliable a classifier is, and the precision-recall curve is… 267 2026-11-13 16:21 H78 hypothesis by AC1: When the compared methods target different functionals, one the mean and another the median, each should be evaluated with a measure consistent for its own tar… [a_priori] 268 2026-11-13 16:21 ST85 status by AC1: H78 -> supported, evidence H37: Each scoring function is lowest in expectation for the functional it is consistent for, so one measure favours the meth… 269 2026-11-13 16:21 H79 hypothesis by AC1: The accuracy of two forecasts of the same series should be compared with the [[TM28:Diebold-Mariano test]]. [a_priori] 270 2026-11-13 16:21 ST86 status by AC1: H79 -> supported, evidence H60: The Diebold-Mariano test allows forecast errors that are serially correlated and not normally distributed, as the error… 271 2026-11-13 16:21 H80 hypothesis by AC1: That a competition or a well-known study used an error measure should not be the reason for choosing it. [a_priori] 272 2026-11-13 16:21 ST87 status by AC1: H80 -> supported, evidence H37: A measure fits a study when it is consistent for what the compared methods predict; that a competition used a measure s… 273 2026-11-13 16:21 H81 hypothesis by AC1: Scale-dependent error measures, such as [[TM5:RMSE]] and [[TM7:MAE]], should not be summed or averaged across series on different scales. [a_priori] 274 2026-11-13 16:21 ST88 status by AC1: H81 -> supported, evidence H51, H52: Scale-dependent measures cannot be compared across series on different scales, and summed or averaged without scaling t… 275 2026-11-13 16:21 H82 hypothesis by AC1: To aggregate the accuracy of point forecasts over series on different scales, each series' errors should first be scaled by a quantity of that series that does… [a_priori] 276 2026-11-13 16:21 ST89 status by AC1: H82 -> supported, evidence H51, H53, H54: Scaled errors are independent of the scale of the data and can be averaged over series; dividing by a quantity of the s… 277 2026-11-13 16:21 H83 hypothesis by AC1: Quantile forecasts should be evaluated with the [[TM12:pinball loss]] at the quantile's level, averaged over the levels when there are several. [a_priori] 278 2026-11-13 16:21 ST90 status by AC1: H83 -> supported, evidence H41: The pinball loss at a level is consistent for the quantile at that level, so its expected value is lowest for the corre… 279 2026-11-13 16:22 H84 hypothesis by AC1: When methods are compared over many series, their accuracy should be summarised by the mean, the median and the quartiles of a scaled measure across the series… [a_priori] 280 2026-11-13 16:22 ST91 status by AC1: H84 -> supported, evidence H53, H57: A scaled measure can be averaged over series, but averages over many data sets are susceptible to outliers; the median… 281 2026-11-13 16:22 H85 hypothesis by AC1: To tell whether several methods whose aggregate accuracy is similar differ significantly over many series or data sets, their ranks on each series should be te… [a_priori] 282 2026-11-13 16:22 ST92 status by AC1: H85 -> supported, evidence H58: The Friedman test, followed by the Nemenyi test, compares methods by their ranks on each data set and assumes only limi… 283 2026-11-13 16:22 H86 hypothesis by AC1: To tell which methods differ significantly from the best one over many series, multiple comparisons with the best ([[TM27:MCB]]) on ranks should be used. [a_priori] 284 2026-11-13 16:22 ST93 status by AC1: H86 -> supported, evidence H59: Multiple comparisons with the best on ranks test which methods differ from the best one, and were used for this on the… 285 2026-11-13 16:22 H87 hypothesis by AC1: When quantile forecasts are compared over series on different scales, each series' [[TM12:pinball loss]] should be scaled by a quantity of that series. [a_priori] 286 2026-11-13 16:22 ST94 status by AC1: H87 -> supported, evidence H55, H41, H51: Pinball losses of series on different scales cannot be compared unscaled; dividing each series' pinball loss by a quant… 287 2026-11-13 16:22 H88 hypothesis by AC1: When prediction intervals are compared over series on different scales, each series' [[TM13:interval score]] should be scaled by a quantity of that series. [a_priori] 288 2026-11-13 16:22 ST95 status by AC1: H88 -> supported, evidence H56, H42, H51: Interval scores of series on different scales cannot be compared unscaled; dividing each series' interval score by a po… 289 2026-11-13 16:23 SC21 screening by AC1: doi:10.18637/jss.v027.i03 full_text: exclude (out of scope: it was cited only for implementation facts of ETS models and back-transformed forecasts, which deci… 290 2026-11-13 16:23 SC22 screening by AC1: url:https://otexts.com/fpp2/transformations.html full_text: exclude (out of scope: it was cited only for implementation facts of ETS models and back-transforme… 291 2026-11-13 16:25 H89 hypothesis by AC1: The [[TM13:interval score]] is a [[TM2:proper scoring rule]] for central prediction intervals that addresses both their width and their coverage. [a_priori] 292 2026-11-13 16:25 ST96 status by AC1: H89 -> supported, evidence XT8, XT9, SR2: Gneiting and Raftery (2007), abstract, Section 1 and Section 6.2. 293 2026-11-13 16:25 H90 hypothesis by AC1: Point forecasts should be evaluated with a [[TM1:consistent scoring function]] for the functional they target, such as the mean, the median or a quantile. [a_priori] 294 2026-11-13 16:25 ST97 status by AC1: H90 -> supported, evidence H37: A forecast is built to target one functional, and only a scoring function consistent for that functional is lowest in e… 295 2026-11-13 16:25 H91 hypothesis by AC1: To tell whether several methods whose aggregate accuracy is similar differ statistically significantly over many series or data sets, their ranks on each serie… [a_priori] 296 2026-11-13 16:25 ST98 status by AC1: H91 -> supported, evidence H58: The Friedman test, followed by the Nemenyi test, compares methods by their ranks on each data set and assumes only limi… 297 2026-11-13 16:25 H92 hypothesis by AC1: To tell which methods differ statistically significantly from the best one over many series, multiple comparisons with the best ([[TM27:MCB]]) on ranks should… [a_priori] 298 2026-11-13 16:25 ST99 status by AC1: H92 -> supported, evidence H59: Multiple comparisons with the best on ranks test which methods differ from the best one, and were used for this on the…