Prediction: forecasting, regression and classification

Version 0.2 (draft, to be reviewed by someone who works in the field) · 2026-11-13

For any study that evaluates predictions: point forecasts, quantiles, intervals or whole predictive distributions, of time series, of a regression's outcome, or of a class.

Principle

A prediction targets something: the mean of the outcome, its median, a quantile, the probability of a class, or the whole distribution. Evaluate it with a scoring function that is consistent for that target, one whose expected value is best when the prediction equals it. A measure that is consistent for a different target rewards a different prediction from the one the method was built to make, and the comparison then says more about the measure than about the methods (Gneiting 2011; Kolassa 2020). So the first question of an evaluation is: what does each compared method predict? The measures follow from the answer.

mean-forecasts

When: the methods give point forecasts of the mean. Check what each implementation returns: the point forecasts of ETS models are conditional means except, beyond one step ahead, with a multiplicative trend or seasonality (Hyndman and Khandakar 2008); forecasts made on a Box-Cox or log scale and back-transformed are medians unless they are bias-adjusted (Hyndman and Athanasopoulos 2018), as in TBATS or ARIMA on a transformed series; models trained on squared error give means.
Default: squared-error measures: RMSE. Across series on different scales, a scaled version: RMSSE (the RMSE scaled by the in-sample one-step naive RMSE, as in the M5 competition; Makridakis, Spiliotis and Assimakopoulos 2022a), or the RMSE relative to a benchmark method.
Why: squared error is consistent for the mean (Gneiting 2011).
Avoid: MAE and MASE as the main measure: they are consistent for the median and reward median forecasts (Gneiting 2011). MAPE: it is consistent for the median of order −1, which lies below the median for positive outcomes, so it rewards forecasts that are too low (Gneiting 2011). sMAPE: it is asymmetric (Goodwin and Lawton 1999) and undefined when actual and forecast are both zero.
References: Gneiting 2011; Makridakis, Spiliotis and Assimakopoulos 2022a; Goodwin and Lawton 1999; Hyndman and Khandakar 2008; Hyndman and Athanasopoulos 2018.

median-forecasts

When: the methods give point forecasts of the median (including back-transformed forecasts without bias adjustment), or the study's question is about typical errors with equal weight for over- and under-forecasting.
Default: absolute-error measures: MAE. Across series on different scales: MASE (Hyndman and Koehler 2006).
Why: absolute error is consistent for the median (Gneiting 2011).
Avoid: with intermittent data (many zeros), the median can be zero, and absolute-error measures then reward a forecast of zero for every period (Kolassa 2016); the M5 competition chose a squared-error measure for this reason (Makridakis, Spiliotis and Assimakopoulos 2022a). Evaluate the mean or the whole distribution as well.
References: Gneiting 2011; Hyndman and Koehler 2006; Kolassa 2016; Makridakis, Spiliotis and Assimakopoulos 2022a.

quantile-forecasts

When: the methods give quantiles (a 90% quantile for a safety stock, several quantiles of a distribution).
Default: the pinball (quantile) loss at each quantile's level, averaged over the levels when there are several; across series on different scales, a scaled pinball loss (as in the M5 uncertainty competition; Makridakis et al. 2022b).
Why: the pinball loss at level τ is consistent for the τ-quantile (Gneiting 2011).
References: Gneiting 2011; Gneiting and Raftery 2007; Makridakis et al. 2022b.

prediction-intervals

When: the methods give prediction intervals.
Default: coverage (the share of outcomes inside the interval, against its nominal level), reported with the interval score, which adds the interval's width and a penalty for each miss.
Why: coverage alone says nothing about width: intervals that are sharp and intervals that are needlessly wide can reach the same coverage. The interval score is a proper scoring rule for central intervals (Gneiting and Raftery 2007).
References: Gneiting and Raftery 2007.

distribution-forecasts

When: the methods give whole predictive distributions (or many quantiles of one).
Default: a proper scoring rule: CRPS, or the log score. Check calibration (the probability integral transform, or coverage at several levels) and report sharpness.
Why: proper scoring rules are best in expectation for the true distribution; calibration and sharpness say how a forecast fails (Gneiting and Raftery 2007; Gneiting, Balabdaoui and Raftery 2007). For counts, see Kolassa 2016.
References: Gneiting and Raftery 2007; Gneiting, Balabdaoui and Raftery 2007; Kolassa 2016.

class-probabilities

When: classifiers that give probabilities of classes.
Default: a proper scoring rule: log loss, or the Brier score. Report calibration (a reliability diagram).
Why: proper scoring rules reward the true probabilities (Brier 1950; Gneiting and Raftery 2007).
Avoid: accuracy, unless the predictions are labels, the costs of errors are equal and the classes balanced. AUC, which measures how well cases are ranked, not whether the probabilities are right; it also weighs the costs of errors differently for different classifiers (Hand 2009). With rare classes, the precision-recall curve can say more than the ROC curve (Saito and Rehmsmeier 2015).
References: Brier 1950; Gneiting and Raftery 2007; Hand 2009; Saito and Rehmsmeier 2015.

same-target

When: the compared methods predict different things (one the mean, another the median).
Default: evaluate each against the target it predicts, or make them predict the same target before comparing. Say in the claim which target the comparison is about.
Why: different error measures are minimised by different summaries of the same predictive distribution, so a single measure favours the methods that target what it is consistent for (Kolassa 2020; Gneiting 2011).
References: Gneiting 2011; Kolassa 2020.

many-series

When: methods are compared over many series or data sets.
Default: report the distribution of each method's scaled errors across series, not only their mean, and test the differences over series: a Friedman test with Nemenyi post-hoc comparisons (Demšar 2006), or its comparison with the best (MCB), used for the M3 competition (Koning et al. 2005); or paired tests on per-series differences with a correction for multiple tests. For two methods on one long series, the Diebold–Mariano test (Diebold and Mariano 1995).
Why: means over many data sets are sensitive to a few outlying ones, and rank-based tests over them do not assume the errors are comparable across them (Demšar 2006).
References: Demšar 2006; Koning et al. 2005; Diebold and Mariano 1995.

precedent

When: a measure is proposed because a competition or a well-known study used it ("the M4 used sMAPE and MASE").
Default: decide by what the compared methods predict (the entries above). Use the precedent's measure as well only to compare with its published results, and say so.
Why: a precedent says what others did, not why it fits this study; the measure has to be consistent for what the methods predict (Gneiting 2011; Kolassa 2020).
References: Gneiting 2011; Kolassa 2020.

References

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30. https://jmlr.org/papers/v7/demsar06a.html
Diebold, F. X., and Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business and Economic Statistics, 13(3), 253–263. doi:10.1080/07350015.1995.10524599
Gneiting, T., Balabdaoui, F., and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society B, 69(2), 243–268. doi:10.1111/j.1467-9868.2007.00587.x
Gneiting, T., and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. doi:10.1198/016214506000001437
Gneiting, T. (2011). Making and evaluating point forecasts. Journal of the American Statistical Association, 106(494), 746–762. doi:10.1198/jasa.2011.r10138
Goodwin, P., and Lawton, R. (1999). On the asymmetry of the symmetric MAPE. International Journal of Forecasting, 15(4), 405–408. doi:10.1016/S0169-2070(99)00007-2
Hand, D. J. (2009). Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine Learning, 77(1), 103–123. doi:10.1007/s10994-009-5119-5
Hyndman, R. J., and Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4), 679–688. doi:10.1016/j.ijforecast.2006.03.001
Hyndman, R. J., and Khandakar, Y. (2008). Automatic time series forecasting: the forecast package for R. Journal of Statistical Software, 27(3), 1–22. doi:10.18637/jss.v027.i03
Hyndman, R. J., and Athanasopoulos, G. (2018). Forecasting: principles and practice (2nd ed.), section 3.2, Transformations and adjustments. OTexts. https://otexts.com/fpp2/transformations.html
Kolassa, S. (2016). Evaluating predictive count data distributions in retail sales forecasting. International Journal of Forecasting, 32(3), 788–803. doi:10.1016/j.ijforecast.2015.12.004
Kolassa, S. (2020). Why the "best" point forecast depends on the error or accuracy measure. International Journal of Forecasting, 36(1), 208–211. doi:10.1016/j.ijforecast.2019.02.017
Koning, A. J., Franses, P. H., Hibon, M., and Stekler, H. O. (2005). The M3 competition: statistical tests of the results. International Journal of Forecasting, 21(3), 397–409. doi:10.1016/j.ijforecast.2004.10.003
Makridakis, S., Spiliotis, E., and Assimakopoulos, V. (2022a). M5 accuracy competition: results, findings, and conclusions. International Journal of Forecasting, 38(4), 1346–1364. doi:10.1016/j.ijforecast.2021.11.013
Makridakis, S., Spiliotis, E., Assimakopoulos, V., Chen, Z., Gaba, A., Tsetlin, I., and Winkler, R. L. (2022b). The M5 uncertainty competition: results, findings and conclusions. International Journal of Forecasting, 38(4), 1365–1385. doi:10.1016/j.ijforecast.2021.10.009
Saito, T., and Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10(3), e0118432. doi:10.1371/journal.pone.0118432

Versions

0.1 (2026-11-13): first draft, for review by someone in the field.
0.2 (2026-11-13): each citation checked against the paper. MAPE's bias rests on Gneiting 2011 (Hyndman and Koehler 2006 state its direction the other way, which Hyndman later corrected); mean forecasts say which models' point forecasts are medians (multiplicative ETS beyond one step, back-transformed forecasts as in TBATS); the M5 papers are cited where their measures are named; the interval, AUC, precision-recall and many-series wording follows what the papers say.

From entrylib/study-planning at 104ba1c (2026-11-13), as the installed claimset package carries it. Changes are proposed there, with the references that support them.