Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4 · C5

D1 MSARIMA analytic interval: The quantiles of predict(interval="prediction") of a pure ARIMA in smooth 1.3.0, which dispatches to the approximate interval. The ARIMA is the state space model $$\begin{aligned} y_t &= w^\top v_{t-l} + \epsilon_t, \\ v_t &= F v_{t-l} + g \epsilon_t, \\ \epsilon_t &\sim N(0, \sigma^2). \end{aligned}$$ The quantile at level $\tau$ and horizon $h$ is $$q_{\tau,h} = \hat y_{T+h|T} + \Phi^{-1}(\tau) \sqrt{V_h}, \qquad V_h = \hat\sigma^2 \frac{T}{T-k} c_h,$$ with $$\hat\sigma^2 = \frac{1}{T} \sum_{t=1}^{T} e_t^2, \qquad c_h = 1 + \sum_{j=1}^{h-1} \left(w^\top F^{j-1} g\right)^2,$$ where $k$ is the number of estimated parameters and $c_h$ the $h$-th diagonal element of the analytic covariance of the state space model, written here for a model with lag 1. Source: Svetunkov (2023).

D8 auto.adam error selection: R auto.adam(y, model="NNN", lags, distribution=c("dnorm","dlnorm"), initial="backcasting") of smooth 4.6: the ARIMA orders and constant are selected by AICc for each distribution with the maxima of auto.msarima (AR (3,3), I (2,1), MA (3,3)), then the distribution with the lower AICc is kept; with dnorm the model is the AutoMSARIMA model, with dlnorm it is the log-normal ARIMA. Quantiles from forecast(interval="prediction"). Source: Svetunkov (2023).

D10 AutoMSARIMA: AutoMSARIMA(lags, initial) in Python and auto.msarima() in R of smooth 1.3.0 / 4.6: ARIMA orders up to AR (3,3), I (2,1), MA (3,3) and a constant selected by AICc, normal errors, the MSARIMA analytic interval for the quantiles. Backcasting initial states unless stated. Source: Svetunkov (2023).

D11 log-normal ARIMA: ADAM with model="NNN", an ARIMA and distribution="dlnorm": the ARIMA applies to the logarithms, $$\begin{aligned} \log y_t &= w^\top v_{t-l} + \log(1+\epsilon_t), \\ v_t &= F v_{t-l} + g \log(1+\epsilon_t), \\ 1+\epsilon_t &\sim \mathrm{logN}(-\sigma^2/2, \sigma^2), \end{aligned}$$ so the scale of the error is proportional to the level. In the comparisons with fixed orders, the orders and constant are those AutoMSARIMA selects on the same series. Source: Svetunkov (2023).

D15 ETS forced and selected: ETS of smooth with backcasting: AutoADAM-style selection among ETS models (ES(model="ZXZ"), or ZXN for non-seasonal series), or ETS(ANN) or ETS(MNN) fitted to every series. Source: Svetunkov (2023).

D2 known-model simulation: Gaussian ARIMA series with $\sigma = 10$ generated from 11 data generating processes (white noise with a constant, random walk with and without drift, AR(1), MA(1), ARIMA(0,1,1), ARIMA(1,1,0), ARIMA(1,1,1), ARIMA(0,2,2), seasonal SARIMA(0,1,1)(0,1,1)$_{12}$ and SARIMA(1,0,0)(1,0,0)$_{12}$) plus an IMA(1,1) with Student-t errors on 4 degrees of freedom; in-sample lengths $T$ of 15, 25, 50, 100 and 300, holdout of 18; 2,000 replicates per process and length with seed 20261105 unless stated.

D5 benchmark: The 5,315 series of M1 (1,001), M3 (3,003) and Tourism (1,311) from fcompdata 0.1.4, with their own holdouts and seasonal periods; lags $(1, m)$ for seasonal series. Members: D18, D19, D20.

D6 M4 sample: 1,500 series each of the yearly, quarterly and monthly M4 series, drawn with seed 20261105 without replacement, with the M4 holdouts. Source: Makridakis et al. (2020).

D9 fresh M4 sample: 1,500 series each of the yearly, quarterly and monthly M4 series, drawn with seed 20261107 from the series not in the M4 sample, so disjoint from it. Source: Makridakis et al. (2020).

D18 M1: The 1,001 series of the M1 competition with their holdouts. Source: Makridakis et al. (1982).

D19 M3: The 3,003 series of the M3 competition with their holdouts. Source: Makridakis and Hibon (2000).

D20 Tourism: The 1,311 series of the tourism forecasting competition with their holdouts. Source: Athanasopoulos et al. (2011).

D4 coverage: For the quantile $q_{\tau,j}$ at level $\tau$ and horizon $j$, the share of holdout observations at or below it, $$c_\tau = \frac{1}{h} \sum_{j=1}^{h} 1\left(y_{T+j} \le q_{\tau,j}\right),$$ averaged first within a series and then over series. cov80 is $c_{0.9} - c_{0.1}$, the coverage of the central 80% interval; q50 is $c_{0.5}$.

D13 MCE: Mean calibration error over the 99 quantile levels $\tau = 0.01, \dots, 0.99$: $$\mathrm{MCE} = \frac{1}{99} \sum_{\tau} \left|\bar c_\tau - \tau\right|,$$ with $\bar c_\tau$ the coverage of $q_\tau$.

D14 scaled pinball: Pinball loss averaged over the 99 quantile levels and the horizons, divided by the mean absolute first difference of the in-sample data for real series; in the simulation, by $\sigma$, and relpin is its ratio to the pinball of the true predictive distribution.

D17 level ratio: Mean of the last third of the in-sample data of a series divided by the mean of all its in-sample data.

D7 level terciles: Series split into terciles of the level ratio across the series of a data set; cov80 reported for each tercile, and the spread as the largest minus the smallest of the three.

D21 smooth 1.0.8 and 1.3.0: Python smooth at tag v1.0.8-py (a separate worktree with its own environment) and at 1.3.0 (commit 634657d8 of openforecast-org/smooth).