The claim
On the M1, M3 and tourism data, CES has the lowest mean RMSSE and the lowest median RMSSE, MASE and MSIS of four methods, CES, ETS, ARIMA and the Theta method, but not the lowest mean MASE, and the highest mean MSIS.
Scope: population: time series of the M1, M3 and tourism forecasting competitions; conditions: seasonal or non-seasonal CES selected by AICc · Data: E2 (T1.r1, T1.r2, T1.r3, T1.r4) · Uses: C34 · Limitations: L1, L3
Claims it rests on
On the M1, M3 and tourism data, in Nemenyi tests at the 5% level, CES has the best mean rank of four methods, CES, ETS, ARIMA and the Theta method on RMSSE and on MASE.
Scope: conditions: Nemenyi test on ranks of the error measures, significance level 0.05 · Data: E2 · Used by: C3
Evidence
CES with automatic seasonal selection compared with ETS, ARIMA, the Theta method, SCUM and SCUM(noCES) on the M1, M3 and tourism data, by error measures on a single holdout per series, with Nemenyi tests of the differences.
Setup: studied CES with automatic seasonal selection, ETS, ARIMA, Theta method, SCUM, SCUM(noCES) · cases M1, M3 and tourism data · procedure holdout at the M1, M3 and tourism horizons, Nemenyi test · lens RMSSE, MASE, MSIS
- R1 MSIS = 523.034 (the six compared methods: CES with automatic seasonal selection, series: M1 series 47)
- R2 MSIS = 25.59 (the six compared methods: ETS, series: M1 series 47, ETS model selected: ETS(M,N,N))
| Row | the six compared methods | RMSSE (mean) | MASE (mean) | MSIS (mean) | RMSSE (median) | MASE (median) | MSIS (median) |
|---|---|---|---|---|---|---|---|
| r1 | CES with automatic seasonal selection | 1.959 | 2.272 | 3.465 | 1.17 | 1.298 | 0.869 |
| r2 | ETS | 1.97 | 2.263 | 2.258 | 1.181 | 1.323 | 0.925 |
| r3 | ARIMA | 2.134 | 2.482 | 3.335 | 1.271 | 1.419 | 0.988 |
| r4 | Theta method | 1.965 | 2.252 | 2.531 | 1.238 | 1.377 | 0.895 |
| r5 | SCUM | 1.867 | 2.146 | 2.333 | 1.128 | 1.256 | 0.844 |
| r6 | SCUM(noCES) | 1.911 | 2.191 | 2.223 | 1.143 | 1.279 | 0.859 |
Limitations
CES produces very large errors on a few series: on M1 series 47 it fails to identify the trend and its MSIS is 523.034 against 25.590 for ETS(M,N,N), which inflates its mean MSIS, and on the non-seasonal subset it fails on two series.
The effect of the length of a time series on the accuracy of CES is not analysed.
Assumptions
Definitions
Methods 10
D1 CES: Complex exponential smoothing: a forecasting method that combines each past observation and its forecast error into the complex variable and forecasts with an exponentially weighted sum of these values, or recursively where is the complex smoothing parameter and carries the information on past forecast errors. Assumes C60, C62, C63, C64.
D4 · a kind of D1 seasonal CES: CES extended with a second pair of level and nonlinear trend states taken at lag , the seasonal period, with its own complex smoothing parameter : ,
D17 · a kind of D1 CES with automatic seasonal selection: CES fitted by auto.ces() from the smooth package 3.1.2, which chooses between seasonal CES and non-seasonal CES by AICc.
D5 SCUM: Simple combination of univariate models: the median of the forecasts of ETS, ARIMA, the Theta method and CES. Source: Petropoulos and Svetunkov (2020).
D6 · a kind of D5 SCUM(noCES): The median of the forecasts of ETS, ARIMA and the Theta method: SCUM without CES.
D7 the six compared methods: The four compared methods and the two combinations, SCUM and SCUM(noCES). Members: D17, D9, D10, D11, D5, D6.
D9 ETS: Exponential smoothing in a state space framework with automatic selection among its models; here es() from the smooth package with model ZXZ, selecting by AICc. Source: Hyndman et al. (2002).
D10 ARIMA: Automatic state space ARIMA; here auto.ssarima() from the smooth package. Source: Svetunkov and Boylan (2020).
D11 Theta method: The Theta method of forecasting; here thetaf() from the forecast package 8.14. Source: Assimakopoulos and Nikolopoulos (2000).
D12 the four compared methods: CES with automatic seasonal selection, ETS, ARIMA and the Theta method, as the benchmark runs them. Members: D17, D9, D10, D11.
Data 1
D8 M1, M3 and tourism data: The 5,315 yearly, quarterly, monthly and other time series of the M1, M3 and tourism forecasting competitions.
Variables and measures 4
D2 complex smoothing parameter: The smoothing parameter of CES, the complex number ; its real part and imaginary part are the two real parameters estimated, and they set how much of the past observations and of the past forecast errors is carried into the forecast.
D14 RMSSE: Root mean squared scaled error: the root mean squared forecast error over the horizon, scaled by the mean absolute first difference of the in-sample data; reported as its mean and median over the series. Source: Makridakis et al. (2022).
D15 MASE: Mean absolute scaled error: the mean absolute forecast error over the horizon, scaled by the mean absolute first difference of the in-sample data; reported as its mean and median over the series. Source: Hyndman and Koehler (2006).
D16 MSIS: Mean scaled interval score: the interval score of a prediction interval averaged over the horizon and scaled by the mean absolute first difference of the in-sample data; here of the 95% prediction interval, reported as its mean and median over the series. Source: Gneiting and Raftery (2007).
Study design 2
D36 holdout at the M1, M3 and tourism horizons: Each series split into an in-sample part and a holdout of the length used in its competition: 6 observations for yearly, 8 for quarterly and other, and 18 for monthly series; forecasts made from the end of the in-sample part.
D38 Nemenyi test: Post hoc Nemenyi test on the ranks of an error measure across series, comparing all methods at the 5% level; here rmcb() from the greybox package. Source: Demšar (2006).
Concepts 1
D3 nonlinear trend component: The state of the CES state-space model, which carries information on both its own previous value and the previous level ; it does not correspond to the trend component of exponential smoothing.
References
- Assimakopoulos, V., Nikolopoulos, K., 2000. The theta model: a decomposition approach to forecasting. International Journal of Forecasting 16, 521–530. https://doi.org/10.1016/S0169-2070(00)00066-2 Used by: D11
- Athanasopoulos, G., Hyndman, R. J., Song, H., Wu, D. C., 2011. The tourism forecasting competition. International Journal of Forecasting 27, 822–844. https://doi.org/10.1016/j.ijforecast.2010.04.009 Used by: A3
- Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1–30. Used by: D38
- Gneiting, T., Raftery, A. E., 2007. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102, 359–378. https://doi.org/10.1198/016214506000001437 Used by: D16
- Hyndman, R. J., Khandakar, Y., 2008. Automatic Time Series Forecasting: The forecast Package for R. Journal of Statistical Software 27. https://doi.org/10.18637/jss.v027.i03 Used by: A5
- Hyndman, R. J., Koehler, A. B., 2006. Another look at measures of forecast accuracy. International Journal of Forecasting 22, 679–688. https://doi.org/10.1016/j.ijforecast.2006.03.001 Used by: D15
- Hyndman, R. J., Koehler, A. B., Snyder, R. D., Grose, S., 2002. A state space framework for automatic forecasting using exponential smoothing methods. International Journal of Forecasting 18, 439–454. https://doi.org/10.1016/S0169-2070(01)00110-8 Used by: D9
- Makridakis, S., Hibon, M., 2000. The M3-Competition: results, conclusions and implications. International Journal of Forecasting 16, 451–476. https://doi.org/10.1016/S0169-2070(00)00057-1 Used by: A2
- Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E., Winkler, R., 1982. The accuracy of extrapolation (time series) methods: Results of a forecasting competition. Journal of Forecasting 1, 111–153. https://doi.org/10.1002/for.3980010202 Used by: A1
- Makridakis, S., Spiliotis, E., Assimakopoulos, V., 2022. M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting 38, 1346–1364. https://doi.org/10.1016/j.ijforecast.2021.11.013 Used by: D14
- Petropoulos, F., Svetunkov, I., 2020. A simple combination of univariate models. International Journal of Forecasting 36, 110–115. https://doi.org/10.1016/j.ijforecast.2019.01.006 Used by: D5
- R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/ Used by: A8
- Svetunkov, I., Boylan, J. E., 2020. State-space ARIMA for supply-chain forecasting. International Journal of Production Research 58, 818–827. https://doi.org/10.1080/00207543.2019.1600764 Used by: D10
- https://github.com/config-i1/greybox Used by: A6
- https://github.com/config-i1/smooth Used by: A4
Artefacts
- A1 · data · Makridakis et al. (1982) M1 competition data
- A2 · data · Makridakis and Hibon (2000) M3 competition data
- A3 · data · Athanasopoulos et al. (2011) Tourism forecasting competition data
- A4 · software · https://github.com/config-i1/smooth smooth package for R (ces, auto.ces, es, auto.ssarima) smooth_3.1.2.tar.gz
- A5 · software · Hyndman and Khandakar (2008) forecast package for R (thetaf) forecast_8.14.tar.gz
- A6 · software · https://github.com/config-i1/greybox greybox package for R (rmcb, Nemenyi test) greybox_0.7.0.tar.gz
- A8 · software · R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/ (2021) R, the environment for statistical computing in which the experiments were run www.R-project.org
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C3: 73, Partly supported
Novelty: new, by the authors
- −10 Evidence Rests on argument only; an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −6 Open problems Limitation L1 (could bias magnitude, not mitigated): CES produces very large errors on a few series: on M1 series 47 it fails to identify the trend and its MSIS is 523.034 against 25.590 for ETS(M,N,N), which inflates its mean MSIS, and on the non-seasonal subset it fails on two series. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −5 Context A principal claim related to no earlier work. To remove it: Relate the claim to the earlier work it extends, addresses or departs from.
- −1 Open problems Limitation L3 (reduces transferability, not mitigated): The effect of the length of a time series on the accuracy of CES is not analysed. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C34: 90, Well supported
Novelty: new, by the authors
- −10 Evidence Rests on argument only; an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).