TRAIL OF H22 (claim C20) Statement: When methods are compared over many series, their accuracy should be summarised by the mean, the median and the quartiles of a scaled measure across the series, such as [[TM6:RMSSE]] or [[TM8:MASE]]. Status: supported In log order: the question, each version of the hypothesis, the plans that name it with their approvals, deviations and runs, its statuses, and the checks, decisions and notes that refer to them. Results are given by number and file. 2026-11-13 13:18 Q1 question by AC1: Which measures and tests should a study use to evaluate and compare predictions (point forecasts of a mean, a median or a quantile, prediction intervals, predi… 2026-11-13 13:18 TM6 term by AC1: RMSSE [variable] 2026-11-13 13:18 TM8 term by AC1: MASE [variable] 2026-11-13 13:20 H22 hypothesis by AC1: When methods are compared over many series or data sets, the distribution of their scaled errors across the series should be reported, not only the mean, becau… [a_priori] 2026-11-13 13:31 ST22 status by AC1: H22 -> supported, evidence SR4: Demsar (2006, SR4, Section 3.1): 'Averages are also susceptible to outliers. They allow classifier's excellent performa… 2026-11-13 14:07 H31 hypothesis by AC1: When methods are compared over many series, their accuracy should be summarised by aggregates of a scaled measure across the series, the mean together with the… [a_priori] 2026-11-13 14:08 ST39 status by AC1: H31 -> supported, evidence SR4, SR7, SR9: Demsar (2006, SR4, Section 3.1): 'Averages are also susceptible to outliers'. Hyndman and Koehler (2006, SR7, Section 3… 2026-11-13 16:18 H53 hypothesis by AC1: Scaled errors, as in [[TM8:MASE]], are independent of the scale of the data and can be averaged over series on different scales. [a_priori] 2026-11-13 16:19 H57 hypothesis by AC1: Averages of a performance measure over many data sets are susceptible to outliers. [a_priori] 2026-11-13 16:22 H84 hypothesis by AC1: When methods are compared over many series, their accuracy should be summarised by the mean, the median and the quartiles of a scaled measure across the series… [a_priori] 2026-11-13 16:22 ST91 status by AC1: H84 -> supported, evidence H53, H57: A scaled measure can be averaged over series, but averages over many data sets are susceptible to outliers; the median…