The claim
On the M4 data without 10 series, SCUM has lower mean and median MASE, RMSSE and MSIS than SCUM(noCES).
Scope: population: M4 competition time series; not for: 10 series on which ETS broke down · Data: E8 (T1.r5, T1.r6) · Limitations: L3, L5
Evidence
The same six methods compared on the M4 data without 10 series by error measures on a single holdout per series, without a significance test.
Setup: studied CES with automatic seasonal selection, ETS, ARIMA, Theta method, SCUM, SCUM(noCES) · cases M4 data without 10 series · procedure holdout at the M4 horizons · lens MASE, RMSSE, MSIS
| Row | the six compared methods | MASE (mean) | RMSSE (mean) | MSIS (mean) | MASE (median) | RMSSE (median) | MSIS (median) |
|---|---|---|---|---|---|---|---|
| r1 | CES with automatic seasonal selection | 2.921 | 2.339 | 2.379 | 1.803 | 1.529 | 0.592 |
| r2 | ETS | 3.31 | 2.688 | 4.751 | 1.831 | 1.542 | 0.632 |
| r3 | ARIMA | 3.037 | 2.436 | 2.272 | 1.903 | 1.606 | 0.617 |
| r4 | Theta method | 2.98 | 2.394 | 2.13 | 1.865 | 1.574 | 0.597 |
| r5 | SCUM | 2.737 | 2.198 | 1.751 | 1.742 | 1.475 | 0.568 |
| r6 | SCUM(noCES) | 2.784 | 2.241 | 1.798 | 1.769 | 1.498 | 0.583 |
Limitations
The effect of the length of a time series on the accuracy of CES is not analysed.
The 10 M4 series on which ETS broke down were excluded from the M4 comparison, so the series were selected on the failure of one of the compared methods.
Assumptions
Definitions
Methods 10
D1 CES: Complex exponential smoothing: a forecasting method that combines each past observation and its forecast error into the complex variable and forecasts with an exponentially weighted sum of these values, or recursively where is the complex smoothing parameter and carries the information on past forecast errors. Assumes C60, C62, C63, C64.
D4 · a kind of D1 seasonal CES: CES extended with a second pair of level and nonlinear trend states taken at lag , the seasonal period, with its own complex smoothing parameter : ,
D17 · a kind of D1 CES with automatic seasonal selection: CES fitted by auto.ces() from the smooth package 3.1.2, which chooses between seasonal CES and non-seasonal CES by AICc.
D5 SCUM: Simple combination of univariate models: the median of the forecasts of ETS, ARIMA, the Theta method and CES. Source: Petropoulos and Svetunkov (2020).
D6 · a kind of D5 SCUM(noCES): The median of the forecasts of ETS, ARIMA and the Theta method: SCUM without CES.
D7 the six compared methods: The four compared methods and the two combinations, SCUM and SCUM(noCES). Members: D17, D9, D10, D11, D5, D6.
D9 ETS: Exponential smoothing in a state space framework with automatic selection among its models; here es() from the smooth package with model ZXZ, selecting by AICc. Source: Hyndman et al. (2002).
D10 ARIMA: Automatic state space ARIMA; here auto.ssarima() from the smooth package. Source: Svetunkov and Boylan (2020).
D11 Theta method: The Theta method of forecasting; here thetaf() from the forecast package 8.14. Source: Assimakopoulos and Nikolopoulos (2000).
D12 the four compared methods: CES with automatic seasonal selection, ETS, ARIMA and the Theta method, as the benchmark runs them. Members: D17, D9, D10, D11.
Data 1
D32 M4 data without 10 series: The 100,000 series of the M4 competition (23,000 yearly, 24,000 quarterly, 48,000 monthly, 359 weekly, 4,227 daily and 414 hourly) except series 19357, 43160, 51552, 62738, 92370, 92435, 93159, 93196, 93412 and 93491, whose influential outliers and structural breaks made ETS break down. Source: Makridakis et al. (2020).
Variables and measures 4
D2 complex smoothing parameter: The smoothing parameter of CES, the complex number ; its real part and imaginary part are the two real parameters estimated, and they set how much of the past observations and of the past forecast errors is carried into the forecast.
D14 RMSSE: Root mean squared scaled error: the root mean squared forecast error over the horizon, scaled by the mean absolute first difference of the in-sample data; reported as its mean and median over the series. Source: Makridakis et al. (2022).
D15 MASE: Mean absolute scaled error: the mean absolute forecast error over the horizon, scaled by the mean absolute first difference of the in-sample data; reported as its mean and median over the series. Source: Hyndman and Koehler (2006).
D16 MSIS: Mean scaled interval score: the interval score of a prediction interval averaged over the horizon and scaled by the mean absolute first difference of the in-sample data; here of the 95% prediction interval, reported as its mean and median over the series. Source: Gneiting and Raftery (2007).
Study design 1
D37 holdout at the M4 horizons: Each series split into an in-sample part and a holdout of the length used in the M4 competition: 6 observations for yearly, 8 for quarterly, 18 for monthly, 13 for weekly, 14 for daily and 48 for hourly series; forecasts made from the end of the in-sample part.
Concepts 1
D3 nonlinear trend component: The state of the CES state-space model, which carries information on both its own previous value and the previous level ; it does not correspond to the trend component of exponential smoothing.
References
- Assimakopoulos, V., Nikolopoulos, K., 2000. The theta model: a decomposition approach to forecasting. International Journal of Forecasting 16, 521–530. https://doi.org/10.1016/S0169-2070(00)00066-2 Used by: D11
- Gneiting, T., Raftery, A. E., 2007. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102, 359–378. https://doi.org/10.1198/016214506000001437 Used by: D16
- Hyndman, R. J., Khandakar, Y., 2008. Automatic Time Series Forecasting: The forecast Package for R. Journal of Statistical Software 27. https://doi.org/10.18637/jss.v027.i03 Used by: A5
- Hyndman, R. J., Koehler, A. B., 2006. Another look at measures of forecast accuracy. International Journal of Forecasting 22, 679–688. https://doi.org/10.1016/j.ijforecast.2006.03.001 Used by: D15
- Hyndman, R. J., Koehler, A. B., Snyder, R. D., Grose, S., 2002. A state space framework for automatic forecasting using exponential smoothing methods. International Journal of Forecasting 18, 439–454. https://doi.org/10.1016/S0169-2070(01)00110-8 Used by: D9
- Makridakis, S., Spiliotis, E., Assimakopoulos, V., 2020. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting 36, 54–74. https://doi.org/10.1016/j.ijforecast.2019.04.014 Used by: D32, A7
- Makridakis, S., Spiliotis, E., Assimakopoulos, V., 2022. M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting 38, 1346–1364. https://doi.org/10.1016/j.ijforecast.2021.11.013 Used by: D14
- Petropoulos, F., Svetunkov, I., 2020. A simple combination of univariate models. International Journal of Forecasting 36, 110–115. https://doi.org/10.1016/j.ijforecast.2019.01.006 Used by: D5
- R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/ Used by: A8
- Svetunkov, I., Boylan, J. E., 2020. State-space ARIMA for supply-chain forecasting. International Journal of Production Research 58, 818–827. https://doi.org/10.1080/00207543.2019.1600764 Used by: D10
- https://github.com/config-i1/smooth Used by: A4
Artefacts
- A4 · software · https://github.com/config-i1/smooth smooth package for R (ces, auto.ces, es, auto.ssarima) smooth_3.1.2.tar.gz
- A5 · software · Hyndman and Khandakar (2008) forecast package for R (thetaf) forecast_8.14.tar.gz
- A7 · data · Makridakis et al. (2020) M4 competition data
- A8 · software · R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/ (2021) R, the environment for statistical computing in which the experiments were run www.R-project.org
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C12: 83, Well supported
Novelty: new, by the authors
- −6 Open problems Limitation L5 (could bias magnitude, not mitigated): The 10 M4 series on which ETS broke down were excluded from the M4 comparison, so the series were selected on the failure of one of the compared methods. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −5 Context A principal claim related to no earlier work. To remove it: Relate the claim to the earlier work it extends, addresses or departs from.
- −1 Open problems Limitation L3 (reduces transferability, not mitigated): The effect of the length of a time series on the accuracy of CES is not analysed. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.