The claim
On the HadCRUT4 Greenland Mean Surface Temperature series and the misspecified Air Passengers series, GTMSE achieves lower out-of-sample RMSE, MAE, and absolute ME than MSE_1.
Scope: conditions: 60-month holdout on Greenland temperature and 12-month holdout on Air Passengers; data: monthly univariate time series · Data: E4 (T1) · Data: E5 (T1) · Uses: C16, C17 · Limitations: L6 · Extends: B11, Section 1, p. 266
Claims it rests on
On the HadCRUT4 Greenland Mean Surface Temperature series, GTMSE, TMSE, and MSCE achieve lower out-of-sample RMSE, MAE, and absolute ME than MSE_1 and General Predictive Likelihood.
Scope: conditions: 60-month holdout window (Jan 2013–Dec 2017) with ARIMA(1,1,2); data: monthly climate time series · Data: E4 (T1) · Limitations: L6 · Used by: C6
On the misspecified Air Passengers series, GTMSE achieves the lowest out-of-sample RMSE, MAE, and ME among all compared estimators.
Scope: conditions: 12-month holdout window with misspecified ETS(A,A,A); data: monthly airline passenger time series · Data: E5 (T1) · Limitations: L6 · Used by: C6
Evidence
Empirical out-of-sample evaluation of ARIMA(1,1,2) estimated with MSE_1, MSE_h (h=60), TMSE, GTMSE, MSCE, and GPL on the HadCRUT.4.5.0.0 monthly Mean Surface Temperature series for Greenland (322.5°, 62.5°), trained on 1,956 observations (Jan 1849–Dec 2012) and tested on 60 holdout observations (Jan 2013–Dec 2017) using smooth v3.2.1 in R (Section 6.1, Table 2).
Setup: studied compared estimators · cases HadCRUT4 Greenland Mean Surface Temperature series · lens RMSE, MAE, ME
| Row | compared estimators | RMSE | MAE | ME | AR(1) parameter | MA(1) parameter | MA(2) parameter |
|---|---|---|---|---|---|---|---|
| r1 | MSE_1 | 1.217 | 0.849 | -0.45 | 0.558 | -1.235 | 0.265 |
| r2 | MSE_h | 1.128 | 0.784 | 0.009 | -0.786 | -0.13 | -0.858 |
| r3 | TMSE | 1.198 | 0.827 | -0.402 | -0.002 | -0.68 | -0.291 |
| r4 | GTMSE | 1.19 | 0.819 | -0.366 | 0.691 | -1.291 | 0.305 |
| r5 | MSCE | 1.185 | 0.817 | -0.343 | 0.634 | -1.059 | 0.077 |
| r6 | General Predictive Likelihood | 1.214 | 0.846 | -0.444 | 0.532 | -1.219 | 0.244 |
Empirical out-of-sample evaluation of a misspecified pure additive ETS(A,A,A) model estimated with MSE_1, MSE_h (h=12), TMSE, GTMSE, MSCE, and GPL on the monthly Air Passengers series over a 12-month forecast horizon using es() in smooth v3.2.1 in R (Section 6.2, Table 3).
Setup: studied compared estimators · cases Air Passengers series · lens RMSE, MAE, ME
| Row | compared estimators | RMSE | MAE | ME | level smoothing parameter alpha | trend smoothing parameter beta | seasonal smoothing parameter gamma |
|---|---|---|---|---|---|---|---|
| r1 | MSE_1 | 49.423 | 37.115 | 28.369 | 1 | 0 | 0 |
| r2 | MSE_h | 77.194 | 57.983 | 54.619 | 0.806 | 0 | 0.089 |
| r3 | TMSE | 19.266 | 15.616 | 9.739 | 0.126 | 0 | 0.798 |
| r4 | GTMSE | 18.672 | 15.005 | 8.369 | 0.145 | 0 | 0.826 |
| r5 | MSCE | 22.601 | 18.736 | 13.922 | 0.102 | 0 | 0.546 |
| r6 | General Predictive Likelihood | 20.014 | 16.426 | 11.128 | 0.111 | 0 | 0.846 |
Limitations
Empirical evaluations are limited to two individual univariate time series (Greenland Mean Surface Temperature and Air Passengers) with single fixed holdout origins rather than a large multi-series benchmark.
Definitions
Methods 7
D2 MSE_1: Conventional one-step-ahead mean squared error estimator minimising where is the one-step-ahead in-sample forecast error over observations.
D3 MSE_h: Multi-step mean squared error estimator minimising the sample squared error at a specific forecast horizon : where is the conditional -step-ahead forecast error. Source: TIAO and XU (1993).
D4 TMSE: Trace Mean Squared Error estimator minimising the sum of squared forecast errors across horizons to : Source: Stern and Coe (1984).
D5 MSCE: Mean Squared Cumulative Error estimator minimising the squared sum of forecast errors over horizons to : Source: doi:10.1016/j.omega.2021.102614.
D6 GTMSE: Geometric Trace Mean Squared Error estimator introduced in this paper, minimising the sum of logarithmic multi-step mean squared errors across horizons to :
D7 General Predictive Likelihood: General Predictive Likelihood for consecutive observations under Gaussian errors, whose concentrated log-likelihood is where is the sample multi-step forecast error covariance matrix.
D15 compared estimators: Group of all six parameter estimators compared in the empirical evaluations: MSE_1, MSE_h, TMSE, GTMSE, MSCE, and General Predictive Likelihood. Members: D2, D3, D4, D6, D5, D7.
Data 2
D13 HadCRUT4 Greenland Mean Surface Temperature series: Monthly Mean Surface Temperature time series from HadCRUT.4.5.0.0 at grid cell in Greenland, January 1849 to December 2017 (2,016 observations total; first 1,956 for training and final 60 for holdout evaluation).
D14 Air Passengers series: Classical monthly airline passenger totals time series from Box and Jenkins (1976) evaluated with a misspecified pure additive model over a 12-month forecast horizon. Source: Box G (1976).
Variables and measures 3
D16 RMSE: Root Mean Squared Error over forecast horizon :
D17 MAE: Mean Absolute Error over forecast horizon :
D18 ME: Mean Error measuring forecast bias over forecast horizon :
Context
References
- Box G, Jenkins G (1976) Time series analysis: forecasting and control. Holden-day, Oakland Used by: D14
- Stern, R. D., Coe, R., 1984. A Model Fitting Analysis of Daily Rainfall Data. Journal of the Royal Statistical Society. Series A (General) 147, 1. https://doi.org/10.2307/2981736 Used by: D4
- TIAO, G. C., XU, D., 1993. Robustness of maximum likelihood estimates for multi-step predictions: The exponential smoothing case. Biometrika 80, 623–641. https://doi.org/10.1093/biomet/80.3.623 Used by: D3
- https://doi.org/10.1016/j.omega.2021.102614 Used by: D5
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C6: 84, Well supported
Novelty: new, by the authors
- −10 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −1 Open problems Limitation L6 (reduces transferability, not mitigated): Empirical evaluations are limited to two individual univariate time series (Greenland Mean Surface Temperature and Air Passengers) with single fixed holdout origins rather than a large multi-series benchmark. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C16: 89, Well supported
Novelty: new, by the authors
- −10 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −1 Open problems Limitation L6 (reduces transferability, not mitigated): Empirical evaluations are limited to two individual univariate time series (Greenland Mean Surface Temperature and Air Passengers) with single fixed holdout origins rather than a large multi-series benchmark. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C17: 89, Well supported
Novelty: new, by the authors
- −10 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −1 Open problems Limitation L6 (reduces transferability, not mitigated): Empirical evaluations are limited to two individual univariate time series (Greenland Mean Surface Temperature and Air Passengers) with single fixed holdout origins rather than a large multi-series benchmark. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.