Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4

Ivan Svetunkov

In brief

94With the true orders and true parameters, the MSARIMA analytic interval is calibrated over the 99 quantiles in the known-model simulation at $T$ of 100 and 300: MCE lies between 0.0007 and 0.0024 and the ratio of the interval width to the theoretical width between 0.9965 and 1.0019, with backcasting and with optimal initial states. C1
83On the M4 sample, the 80% coverage of AutoMSARIMA falls with the level ratio: 0.813, 0.700 and 0.554 from the lowest to the highest of the level terciles, while ETS(MNN) of ETS forced and selected, whose error scale follows the level, covers 0.764, 0.778 and 0.814. C2
78On the M4 sample, the log-normal ARIMA with the orders AutoMSARIMA selects removes the fall of 80% coverage over the level terciles (0.774, 0.783 and 0.752, a spread of 0.031) and lowers MCE from 0.0386 to 0.0177. C3
79On the fresh M4 sample, auto.adam error selection lowers MCE from 0.0352 to 0.0224 and the spread of 80% coverage over the level terciles from 0.272 to 0.077 against R auto.msarima (AutoMSARIMA), short of the planned reduction of MCE to 0.6 times that of auto.msarima, so the planned test is inconclusive. C5

Context

Addresses Svetunkov (2023) · from C1: The book derives the interval; this checks the implementation of smooth 1.3.0 against it.
Addresses Chatfield (1993) · from C2: Chatfield discusses why intervals are too narrow in general; this locates the shortfall of one implementation on series whose level has grown.
Addresses Svetunkov (2023) · from C7: Checks that the analytic variance of the implementation matches simulation from the same model.
Extends Chatfield (1993) · from C13: Chatfield lists model selection among the reasons intervals are too narrow; this measures it for AutoMSARIMA.
Extends Chatfield (1993) · from C14: Chatfield attributes narrow intervals partly to ignored parameter uncertainty; this sizes the effect at T = 100.
Addresses Svetunkov (2023) · from C15: The book proposes reforecasting to include parameter uncertainty; this tests the implementation for ARIMA.
Extends Chatfield (1993) · from C21: Sizes the effect of estimation and selection on short series for one implementation.

Definitions

Methods 7

D1 MSARIMA analytic interval: The quantiles of predict(interval="prediction") of a pure ARIMA in smooth 1.3.0, which dispatches to the approximate interval. The ARIMA is the state space model $\begin{aligned} y_t &= w^\top v_{t-l} + \epsilon_t, \\ v_t &= F v_{t-l} + g \epsilon_t, \\ \epsilon_t &\sim N(0, \sigma^2). \end{aligned}$ The quantile at level $\tau$ and horizon $h$ is $q_{\tau,h} = \hat y_{T+h|T} + \Phi^{-1}(\tau) \sqrt{V_h}, \qquad V_h = \hat\sigma^2 \frac{T}{T-k} c_h,$ with $\hat\sigma^2 = \frac{1}{T} \sum_{t=1}^{T} e_t^2, \qquad c_h = 1 + \sum_{j=1}^{h-1} \left(w^\top F^{j-1} g\right)^2,$ where $k$ is the number of estimated parameters and $c_h$ the $h$ -th diagonal element of the analytic covariance of the state space model, written here for a model with lag 1. Source: Svetunkov (2023).

D8 auto.adam error selection: R auto.adam(y, model="NNN", lags, distribution=c("dnorm","dlnorm"), initial="backcasting") of smooth 4.6: the ARIMA orders and constant are selected by AICc for each distribution with the maxima of auto.msarima (AR (3,3), I (2,1), MA (3,3)), then the distribution with the lower AICc is kept; with dnorm the model is the AutoMSARIMA model, with dlnorm it is the log-normal ARIMA. Quantiles from forecast(interval="prediction"). Source: Svetunkov (2023).

D10 AutoMSARIMA: AutoMSARIMA(lags, initial) in Python and auto.msarima() in R of smooth 1.3.0 / 4.6: ARIMA orders up to AR (3,3), I (2,1), MA (3,3) and a constant selected by AICc, normal errors, the MSARIMA analytic interval for the quantiles. Backcasting initial states unless stated. Source: Svetunkov (2023).

D11 log-normal ARIMA: ADAM with model="NNN", an ARIMA and distribution="dlnorm": the ARIMA applies to the logarithms, $\begin{aligned} \log y_t &= w^\top v_{t-l} + \log(1+\epsilon_t), \\ v_t &= F v_{t-l} + g \log(1+\epsilon_t), \\ 1+\epsilon_t &\sim \mathrm{logN}(-\sigma^2/2, \sigma^2), \end{aligned}$ so the scale of the error is proportional to the level. In the comparisons with fixed orders, the orders and constant are those AutoMSARIMA selects on the same series. Source: Svetunkov (2023).

D12 reforecast: reforecast() of smooth: draws the parameters 300 times from a normal distribution with the estimated covariance of the parameters, reapplies the model with each draw and simulates the predictive distribution, so that the quantiles include the uncertainty of the parameters.

D15 ETS forced and selected: ETS of smooth with backcasting: AutoADAM-style selection among ETS models (ES(model="ZXZ"), or ZXN for non-seasonal series), or ETS(ANN) or ETS(MNN) fitted to every series. Source: Svetunkov (2023).

D16 other ARIMA implementations: statsforecast AutoARIMA, pmdarima auto_arima and skforecast Arima, as run in the notebook of the smooth benchmark of 2026-11-05, whose saved forecasts were read.

Data 7

D2 known-model simulation: Gaussian ARIMA series with $\sigma = 10$ generated from 11 data generating processes (white noise with a constant, random walk with and without drift, AR(1), MA(1), ARIMA(0,1,1), ARIMA(1,1,0), ARIMA(1,1,1), ARIMA(0,2,2), seasonal SARIMA(0,1,1)(0,1,1) $_{12}$ and SARIMA(1,0,0)(1,0,0) $_{12}$ ) plus an IMA(1,1) with Student-t errors on 4 degrees of freedom; in-sample lengths $T$ of 15, 25, 50, 100 and 300, holdout of 18; 2,000 replicates per process and length with seed 20261105 unless stated.

D5 benchmark: The 5,315 series of M1 (1,001), M3 (3,003) and Tourism (1,311) from fcompdata 0.1.4, with their own holdouts and seasonal periods; lags $(1, m)$ for seasonal series. Members: D18, D19, D20.

D6 M4 sample: 1,500 series each of the yearly, quarterly and monthly M4 series, drawn with seed 20261105 without replacement, with the M4 holdouts. Source: Makridakis et al. (2020).

D9 fresh M4 sample: 1,500 series each of the yearly, quarterly and monthly M4 series, drawn with seed 20261107 from the series not in the M4 sample, so disjoint from it. Source: Makridakis et al. (2020).

D18 M1: The 1,001 series of the M1 competition with their holdouts. Source: Makridakis et al. (1982).

D19 M3: The 3,003 series of the M3 competition with their holdouts. Source: Makridakis and Hibon (2000).

D20 Tourism: The 1,311 series of the tourism forecasting competition with their holdouts. Source: Athanasopoulos et al. (2011).

Variables and measures 4

D4 coverage: For the quantile $q_{\tau,j}$ at level $\tau$ and horizon $j$ , the share of holdout observations at or below it, $c_\tau = \frac{1}{h} \sum_{j=1}^{h} 1\left(y_{T+j} \le q_{\tau,j}\right),$ averaged first within a series and then over series. cov80 is $c_{0.9} - c_{0.1}$ , the coverage of the central 80% interval; q50 is $c_{0.5}$ .

D13 MCE: Mean calibration error over the 99 quantile levels $\tau = 0.01, \dots, 0.99$ : $\mathrm{MCE} = \frac{1}{99} \sum_{\tau} \left|\bar c_\tau - \tau\right|,$ with $\bar c_\tau$ the coverage of $q_\tau$ .

D14 scaled pinball: Pinball loss averaged over the 99 quantile levels and the horizons, divided by the mean absolute first difference of the in-sample data for real series; in the simulation, by $\sigma$ , and relpin is its ratio to the pinball of the true predictive distribution.

D17 level ratio: Mean of the last third of the in-sample data of a series divided by the mean of all its in-sample data.

Study design 2

D3 cases T, E, S, W: What the fitted AutoMSARIMA or MSARIMA knows about the data generating process of the known-model simulation: T, true orders and true AR, MA and constant parameters with initial states and scale estimated; E, true orders with estimated parameters; S, orders selected by AutoMSARIMA; W, a fixed wrong model for each process. Each with backcasting or optimal initial states.

D7 level terciles: Series split into terciles of the level ratio across the series of a data set; cov80 reported for each tercile, and the spread as the largest minus the smallest of the three.

Settings 1

D21 smooth 1.0.8 and 1.3.0: Python smooth at tag v1.0.8-py (a separate worktree with its own environment) and at 1.3.0 (commit 634657d8 of openforecast-org/smooth).

Assumptions

C24 · methodological · unchecked

The coverage of a set of series is well summarised by first averaging the holdout observations within each series and then averaging over series, all horizons together. Rested on by C1.

C25 · data · unchecked

The ratio of the mean of the last third of a series to its overall mean indexes how much the scale of its errors at the forecast origin exceeds the average scale over the sample. Rested on by C2, C8.

Principal claims

94C1 · descriptive · planned test · novelty: new, by the authors · no close match on EntryLib

With the true orders and true parameters, the MSARIMA analytic interval is calibrated over the 99 quantiles in the known-model simulation at $T$ of 100 and 300: MCE lies between 0.0007 and 0.0024 and the ratio of the interval width to the theoretical width between 0.9965 and 1.0019, with backcasting and with optimal initial states.

Scope: data: simulated Gaussian ARIMA and SARIMA series; conditions: holdout of 18, smooth 1.3.0 (Python), case T of the cases T, E, S, W, T of 100 and 300 · Simulation: E1 (T1.r3, T1.r2, T1.r1), artefacts A3, A8 · Uses: C7, C12 · Assumes: C24 · Addresses: Svetunkov (2023) · Planned: lab plan PL4, planned 5 November 2026 · Outcome: computed from the recorded results

83C2 · descriptive · planned test · novelty: new, by the authors · no close match on EntryLib

On the M4 sample, the 80% coverage of AutoMSARIMA falls with the level ratio: 0.813, 0.700 and 0.554 from the lowest to the highest of the level terciles, while ETS(MNN) of ETS forced and selected, whose error scale follows the level, covers 0.764, 0.778 and 0.814.

Scope: data: M4 yearly, quarterly and monthly series; conditions: backcasting, smooth 1.3.0 (Python) · Data: E8 (T1.r1, T1.r2) · Uses: C8 · Assumes: C25 · Limitations: L2, L3 · Addresses: Chatfield (1993) · Planned: lab plan PL12, planned 5 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

78C3 · descriptive · planned test · novelty: new, by the authors · no close match on EntryLib

On the M4 sample, the log-normal ARIMA with the orders AutoMSARIMA selects removes the fall of 80% coverage over the level terciles (0.774, 0.783 and 0.752, a spread of 0.031) and lowers MCE from 0.0386 to 0.0177.

Scope: data: M4 yearly, quarterly and monthly series; conditions: backcasting, orders fixed at those AutoMSARIMA selects · Data: E9 (T1.r2, T1.r1) · Uses: C2 · Limitations: L2 · Planned: lab plan PL13, planned 5 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

73C4 · descriptive · planned test · novelty: new, by the authors · no close match on EntryLib

On known models, the AutoMSARIMA interval of smooth 1.3.0 is calibrated no worse than that of 1.0.8 (smooth 1.0.8 and 1.3.0): at $T = 100$ in the known-model simulation its MCE is 0.0100 against 0.0120 with estimated parameters, 0.0139 against 0.0172 with selected orders and 0.0085 against 0.0246 with optimal initial states.

Scope: data: simulated Gaussian ARIMA and SARIMA series; conditions: holdout of 18, T = 100, cases E and S · Simulation: E10 (T1.r1, T1.r5, T1.r3, T1.r7, T1.r2, T1.r6), artefacts A4, A8 · Uses: C9 · Limitations: L2, L4 · Planned: lab plan PL11, planned 5 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

79C5 · descriptive · planned test · novelty: new, by the authors · no close match on EntryLib

On the fresh M4 sample, auto.adam error selection lowers MCE from 0.0352 to 0.0224 and the spread of 80% coverage over the level terciles from 0.272 to 0.077 against R auto.msarima (AutoMSARIMA), short of the planned reduction of MCE to 0.6 times that of auto.msarima, so the planned test is inconclusive.

Scope: data: M4 yearly, quarterly and monthly series; conditions: backcasting, R smooth 4.6 · Data: E12 (T1.r1, T1.r2), artefacts A6 · Limitations: L5, L6 · Planned: lab plan PL15, planned 9 November 2026 · Outcome: computed from the recorded results

60C6 · interpretive · novelty: new, by the authors · no close match on EntryLib

The wider intervals of ETS than of AutoMSARIMA on the benchmark come from the multiplicative-error models in the ETS pool of ETS forced and selected, not from how either package computes the interval.

Scope: data: M1, M3 and Tourism series; conditions: backcasting · Argument: W1 · Uses: C10, C11

Supporting claims

97C7 · descriptive · planned test · novelty: new, by the authors

With estimated parameters in the known-model simulation, the MSARIMA analytic interval and the simulated interval of the same fit have the same width and MCE at $T$ of 50, 100 and 300 (widths 0.957 and 0.957, 0.981 and 0.980, 0.997 and 0.996; MCE 0.0196 and 0.0197, 0.0100 and 0.0100, 0.0026 and 0.0027).

Scope: data: simulated Gaussian ARIMA and SARIMA series; conditions: holdout of 18, smooth 1.3.0 (Python), case E, backcasting · Simulation: E3 (T1.r1, T1.r2, T1.r3, T1.r4, T1.r5, T1.r6), artefacts A8 · Addresses: Svetunkov (2023) · Planned: lab plan PL6, planned 5 November 2026 · Outcome: computed from the recorded results · Used by: C1

97C8 · descriptive · novelty: new, by the authors

On the 5,315 benchmark series, the fall of 80% coverage over the level terciles is shared by AutoMSARIMA (0.786, 0.668, 0.590) and the other ARIMA implementations: statsforecast AutoARIMA (0.789, 0.667, 0.589), pmdarima (0.770, 0.635, 0.582) and skforecast (0.781, 0.658, 0.585).

Scope: data: M1, M3 and Tourism series; conditions: backcasting for smooth · Data: E13 (T1.r1, T1.r2, T1.r3, T1.r4) · Assumes: C25 · Used by: C2

100C9 · descriptive · novelty: new, by the authors

The width of the AutoMSARIMA interval on the benchmark did not change between smooth 1.0.8 and 1.3.0: the median ratio of the 80% widths of 1.3.0 over 1.0.8 is 1.000 with backcasting and with optimal initial states.

Scope: data: M1, M3 and Tourism series · Data: E17 (T1.r1, T1.r2) · Used by: C4

95C10 · descriptive · novelty: new, by the authors

On the 114 benchmark series where the selected ETS(ANN) of ETS forced and selected and the ARIMA(0,1,1) selected by AutoMSARIMA give the same forecast and width, the two have the same 80% coverage, 0.734.

Scope: data: M1, M3 and Tourism series; conditions: backcasting · Data: E13 (T2.r1, T2.r2, T2.r3) · Limitations: L7 · Used by: C6

100C11 · descriptive · novelty: new, by the authors

Forced on all 5,315 benchmark series, ETS(MNN) of ETS forced and selected has an 80% coverage of 0.764, against 0.668 for ETS(ANN) and 0.695 for ARIMA(0,1,1).

Scope: data: M1, M3 and Tourism series; conditions: backcasting · Data: E14 (T1.r3, T1.r2, T1.r1) · Used by: C6

100C12 · descriptive

R and Python give the same AutoMSARIMA interval on 500 benchmark series: the median ratio of the 80% widths is 1.000, the same model is selected on every series and both have an 80% coverage of 0.677.

Scope: data: 500 M1, M3 and Tourism series; conditions: backcasting · Data: E18 (T1.r1) · Used by: C1

97C13 · descriptive · extends earlier work · planned test · novelty: extends earlier work, by the authors

Whether selecting the orders with AutoMSARIMA loses little coverage in the known-model simulation at $T = 100$ was not settled: pooled cov80 with selected orders was 0.756, between the threshold for little loss (0.77) and that for a large loss (0.74).

Scope: data: simulated Gaussian ARIMA and SARIMA series; conditions: holdout of 18, smooth 1.3.0 (Python), case S, T = 100, backcasting · Simulation: E2 (T1.r1), artefacts A8 · Extends: Chatfield (1993) · Planned: lab plan PL5, planned 5 November 2026 · Outcome: computed from the recorded results

97C14 · descriptive · extends earlier work · planned test · novelty: extends earlier work, by the authors

Estimating the parameters does not cost 6 points of 80% coverage of the MSARIMA analytic interval at $T = 100$ in the known-model simulation: with the true orders, pooled cov80 is 0.800 with true parameters and 0.770 with estimated parameters, a loss of 3 points.

Scope: data: simulated Gaussian ARIMA and SARIMA series; conditions: holdout of 18, smooth 1.3.0 (Python), cases T and E, T = 100, backcasting · Simulation: E4 (T1.r2, T1.r1), artefacts A8 · Extends: Chatfield (1993) · Planned: lab plan PL7, planned 5 November 2026 · Outcome: computed from the recorded results

97C15 · descriptive · planned test · novelty: new, by the authors

reforecast does not improve on the MSARIMA analytic interval of predict() in the known-model simulation at $T = 100$ with estimated parameters: with backcasting its MCE is 0.0265 against 0.0100 and its relative scaled pinball 1.958 against 1.043; with optimal initial states its MCE is lower (0.0053 against 0.0085) but its relative pinball higher (1.396 against 1.058).

Scope: data: simulated Gaussian ARIMA and SARIMA series; conditions: holdout of 18, smooth 1.3.0 (Python), case E, T = 100 · Simulation: E5 (T1.r1, T1.r2, T1.r3, T1.r4), artefacts A8 · Addresses: Svetunkov (2023) · Planned: lab plan PL8, planned 5 November 2026 · Outcome: computed from the recorded results

85C16 · descriptive · planned test · novelty: new, by the authors

A fixed wrong model alone produces under-coverage of the size seen on the benchmark: over 12 wrong models in the known-model simulation at $T = 100$ the pooled 80% coverage of the MSARIMA analytic interval is 0.676 and its q50 0.428.

Scope: data: simulated Gaussian ARIMA and SARIMA series; conditions: holdout of 18, smooth 1.3.0 (Python), case W with 12 fixed wrong models, T = 100, backcasting · Simulation: E6 (T1.r1), artefacts A8 · Limitations: L1 · Planned: lab plan PL9, planned 5 November 2026 · Outcome: computed from the recorded results

97C17 · descriptive · planned test · novelty: new, by the authors

On the 5,315 benchmark series, reforecast does not close the gap of the 80% coverage of AutoMSARIMA: cov80 moves from 0.681 to 0.683 and its mean scaled pinball is 7e191 against 0.899.

Scope: data: M1, M3 and Tourism series; conditions: backcasting, smooth 1.3.0 (Python) · Data: E7 (T1.r1, T1.r2) · Planned: lab plan PL10, planned 5 November 2026 · Outcome: computed from the recorded results

97C18 · descriptive · planned test

The planned test of auto.adam error selection against auto.msarima on the fresh M4 sample under plan PL14 gave no result: its only run stopped on loading the M4 data, before any model was fitted.

Scope: data: M4 yearly, quarterly and monthly series · Data: E11 · Planned: lab plan PL14, planned 7 November 2026 · Outcome: computed from the recorded results

100C19 · descriptive · novelty: new, by the authors

On the 5,315 benchmark series, the log-normal ARIMA with the selected orders raises the 80% coverage of AutoMSARIMA from 0.681 to 0.756 and lowers MCE from 0.0402 to 0.0124, but raises the median scaled pinball from 0.496 to 0.551.

Scope: data: M1, M3 and Tourism series; conditions: backcasting, orders fixed at those AutoMSARIMA selects · Data: E15 (T1.r1, T1.r2)

98C20 · descriptive · novelty: new, by the authors

On the 5,315 benchmark series, auto.adam error selection chooses dlnorm for 64% of the series, raises 80% coverage from 0.681 to 0.729, lowers MCE from 0.0402 to 0.0187 and raises the median scaled pinball from 0.496 to 0.514 against R auto.msarima (AutoMSARIMA).

Scope: data: M1, M3 and Tourism series; conditions: backcasting, R smooth 4.6 · Data: E16 (T1.r1, T1.r2), artefacts A5 · Limitations: L5, L6

83C21 · predictive · extends earlier work · novelty: extends earlier work, by the authors

Estimation and selection alone, as measured in the known-model simulation, predict an 80% coverage of 0.717 for AutoMSARIMA at the lengths of the benchmark series, against 0.801 with the true parameters.

Scope: data: simulated series at the lengths of the M1, M3 and Tourism series; conditions: backcasting, case S · Data: E19 (T1.r3, T1.r1) · Limitations: L8 · Extends: Chatfield (1993)

100C22 · descriptive · novelty: new, by the authors

On the fresh M4 sample, auto.adam error selection has a higher MCE than R auto.msarima (AutoMSARIMA) in the lowest and highest level terciles (0.041 against 0.035, and 0.105 against 0.076) and a lower one only in the middle tercile (0.020 against 0.041), so its lower overall MCE comes in part from errors of opposite sign in different series.

Scope: data: M4 yearly, quarterly and monthly series; conditions: backcasting, R smooth 4.6 · Data: E20 (T4.r3, T4.r4, T4.r5, T4.r6, T4.r7, T4.r8), artefacts A6

100C23 · descriptive · novelty: new, by the authors

On the 5,315 benchmark series, the MCE of AutoMSARIMA is 0.0402 with smooth 1.3.0 against 0.0326 with 1.0.8 (smooth 1.0.8 and 1.3.0) with backcasting.

Scope: data: M1, M3 and Tourism series; conditions: backcasting · Data: E20 (T2.r1, T2.r2), artefacts A4

Evidence

E1 · simulation · cited by C1

Simulation of the known-model simulation: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Case T of cases T, E, S, W with backcasting and optimal initial states (lab run RN6, plan PL4). Width is the ratio of the 80% interval width to the theoretical $h$ -step width; maxcellmce is the largest MCE over the processes.

Setup: studied MSARIMA analytic interval · cases known-model simulation · procedure cases T, E, S, W · lens coverage, MCE

E1.T1 · Case T: the true orders with the true parameters, pooled over the 11 Gaussian processes
Row	T	cases T, E, S, W	initial states	coverage (nominal level: 0.8)	ratio of the 80% interval width to the theoretical width	MCE	largest MCE over the 11 processes
r1	100	T	backcasting	0.7997	1.0019	0.0023	0.0145
r2	100	T	optimal	0.7972	0.9965	0.0024	0.0145
r3	300	T	backcasting		1.0007	0.0007	0.0059
r4	300	T	optimal		0.9992	0.0009	0.0059

Calibration curves of the 99 quantiles in the known-model simulation, cases T, E and S, at T = 15, 25, 50, 100, 300: coverage of each quantile (top) and covera… — A3 Calibration curves of the 99 quantiles in the known-model simulation, cases T, E and S, at T = 15, 25, 50, 100, 300: coverage of each quantile (top) and coverage minus level (bottom).

E2 · simulation · cited by C13

Simulation of the known-model simulation: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Case S, orders selected by AutoMSARIMA (run RN11, plan PL5).

Setup: studied AutoMSARIMA · cases known-model simulation · procedure cases T, E, S, W · lens coverage

E2.T1 · Case S: orders selected by AutoMSARIMA, pooled over the 11 Gaussian processes
Row	T	cases T, E, S, W	initial states	coverage (nominal level: 0.8)
r1	100	S	backcasting	0.7559
r2	100	S	optimal	0.7572

E3 · simulation · cited by C7

Simulation of the known-model simulation: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Case E with the MSARIMA analytic interval against the simulated interval of the same fit (interval="simulated", 10,000 paths) (run RN7, plan PL6).

Setup: studied MSARIMA analytic interval · cases known-model simulation · procedure cases T, E, S, W · lens coverage, MCE

E3.T1 · Case E: the analytic interval against the simulated interval of the same fit, pooled over the 11 Gaussian processes
Row	T	cases T, E, S, W	initial states	interval	ratio of the 80% interval width to the theoretical width	MCE
r1	50	E	backcasting	analytic	0.9572	0.0196
r2	50	E	backcasting	simulated interval	0.957	0.0197
r3	100	E	backcasting	analytic	0.9807	0.01
r4	100	E	backcasting	simulated interval	0.9805	0.01
r5	300	E	backcasting	analytic	0.9966	0.0026
r6	300	E	backcasting	simulated interval	0.9964	0.0027

E4 · simulation · cited by C14

Simulation of the known-model simulation: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Cases T and E (run RN8, plan PL7).

Setup: studied MSARIMA analytic interval · cases known-model simulation · procedure cases T, E, S, W · lens coverage

E4.T1 · Cases T and E, pooled over the 11 Gaussian processes
Row	T	cases T, E, S, W	initial states	coverage (nominal level: 0.8)
r1	100	E	backcasting	0.7701
r2	100	T	backcasting	0.7997

E5 · simulation · cited by C15

Simulation of the known-model simulation: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Case E with predict() against reforecast (suffix _ref) (run RN9, plan PL8).

Setup: studied MSARIMA analytic interval, reforecast · cases known-model simulation · procedure cases T, E, S, W · lens MCE, scaled pinball

E5.T1 · Case E with predict() and with reforecast, pooled over the 11 Gaussian processes
Row	T	cases T, E, S, W	initial states	forecast	MCE	scaled pinball relative to the true model with true parameters
r1	100	E	backcasting	predict()	0.01	1.0432
r2	100	E	backcasting	reforecast	0.0265	1.9575
r3	100	E	optimal	predict()	0.0085	1.0578
r4	100	E	optimal	reforecast	0.0053	1.3961

E6 · simulation · cited by C16

Simulation of the known-model simulation: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Case W: each process fitted with one fixed wrong model (e.g. a random walk with drift fitted as ARIMA(0,1,0) without drift, an ARIMA(1,1,1) as ARIMA(0,1,1)); 12 wrong models chosen by the authors (run RN10, plan PL9).

Setup: studied MSARIMA analytic interval · cases known-model simulation · procedure cases T, E, S, W · lens coverage

E6.T1 · Case W (fixed wrong models) and the true model with true parameters, pooled over the 11 Gaussian processes
Row	T	cases T, E, S, W	coverage (nominal level: 0.8)	share of holdout observations at or below the 0.5 quantile
r1	100	W	0.6756	0.4281
r2	100	true model, true parameters	0.8031	0.4988

E7 · quant · benchmark · cited by C17

AutoMSARIMA with backcasting fitted to each of the 5,315 benchmark series; predict() and reforecast quantiles scored on the holdout, with the harness of the smooth benchmark notebook of 2026-11-05; the predict() coverage reproduces the saved benchmark array exactly (run RN12, plan PL10).

Setup: studied AutoMSARIMA, reforecast · cases benchmark · lens coverage, scaled pinball

E7.T1 · AutoMSARIMA with backcasting on the 5,315 benchmark series: predict() against reforecast (data: benchmark, initial states: backcasting)
Row	forecast	coverage (nominal level: 0.8)	scaled pinball (mean)
r1	predict()	0.6812	0.8991
r2	reforecast	0.6827	6.983616214609794e+191

E8 · quant · benchmark · cited by C2

AutoMSARIMA with backcasting and ETS(MNN) fitted to every series of the M4 sample; cov80 per level terciles (run RN41, plan PL12).

Setup: studied AutoMSARIMA, ETS forced and selected · cases M4 sample · procedure level terciles · lens coverage, MCE, level ratio

E8.T1 · AutoMSARIMA and ETS(MNN) on the M4 sample (data: M4 sample)
Row	method	coverage (nominal level: 0.8)	MCE	coverage (nominal level: 0.8, level terciles: low)	coverage (nominal level: 0.8, level terciles: mid)	coverage (nominal level: 0.8, level terciles: high)	80% coverage in the low tercile minus that in the high tercile	share of holdout observations at or below the 0.5 quantile
r1	AutoMSARIMA	0.6889	0.0386	0.8128	0.6995	0.5544	0.2585
r2	ETS(MNN)	0.7854	0.0848	0.764	0.7779	0.8142	-0.0502	0.3723

E9 · quant · benchmark · cited by C3

AutoMSARIMA and the log-normal ARIMA with the same selected orders on the M4 sample; MCE and cov80 per level terciles (run RN42, plan PL13).

Setup: studied AutoMSARIMA, log-normal ARIMA · cases M4 sample · procedure level terciles · lens coverage, MCE, scaled pinball, level ratio

E9.T1 · AutoMSARIMA and the log-normal ARIMA with the same orders on the M4 sample (data: M4 sample)
Row	method	MCE	scaled pinball (median)	range of the 80% coverage over the level terciles	coverage (nominal level: 0.8)	coverage (nominal level: 0.8, level terciles: low)	coverage (nominal level: 0.8, level terciles: mid)	coverage (nominal level: 0.8, level terciles: high)
r1	AutoMSARIMA	0.0386	0.6752	0.2585
r2	log-normal ARIMA	0.0177	0.7679	0.0311	0.7698	0.7742	0.7832	0.7521

E10 · simulation · cited by C4

Simulation of the known-model simulation: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Cases T, E, S with backcasting and E with optimal initial states, run with smooth 1.0.8 and 1.3.0 on the same seeds (runs RN43 for 1.0.8 and RN44 for 1.3.0, plan PL11).

Setup: studied AutoMSARIMA · cases known-model simulation · procedure cases T, E, S, W · lens coverage, MCE · context smooth 1.0.8 and 1.3.0

E10.T1 · smooth 1.0.8 against 1.3.0 on the same seeds, pooled over the 11 Gaussian processes
Row	T	cases T, E, S, W	initial states	smooth 1.0.8 and 1.3.0	MCE	ratio of the 80% interval width to the theoretical width	coverage (nominal level: 0.8)
r1	100	E	backcasting	1.0.8	0.012
r2	100	E	optimal	1.0.8	0.0246	3.3857
r3	100	S	backcasting	1.0.8	0.0172
r4	100	T	backcasting	1.0.8		1.1105	0.8256
r5	100	E	backcasting	1.3.0	0.01
r6	100	E	optimal	1.3.0	0.0085	1.0111
r7	100	S	backcasting	1.3.0	0.0139
r8	100	T	backcasting	1.3.0		1.0019	0.7997

A4 Calibration curves of the 99 quantiles of AutoMSARIMA 1.0.8 and 1.3.0 on the benchmark (backcasting, optimal) and in the simulation at T = 100.

E11 · quant · benchmark · cited by C18

Planned comparison of auto.adam error selection with auto.msarima on the fresh M4 sample (plan PL14). Its only run, RN47, failed while loading M4 because the cached M4 files had been removed; no model was fitted and no result exists.

Setup: studied auto.adam error selection, AutoMSARIMA · cases fresh M4 sample · procedure level terciles · lens MCE

E12 · quant · benchmark · cited by C5

R auto.msarima (AutoMSARIMA) and auto.adam error selection fitted to the 4,500 series of the fresh M4 sample with backcasting; 99 quantiles per series scored on the M4 holdouts; MCE, cov80 by level terciles, scaled pinball, and the share of series choosing dlnorm (run RN48, plan PL15, which repeats PL14 after its failed run).

Setup: studied auto.adam error selection, AutoMSARIMA · cases fresh M4 sample · procedure level terciles · lens coverage, MCE, scaled pinball, level ratio

E12.T1 · R auto.msarima and auto.adam error selection on the fresh M4 sample, backcasting (data: fresh M4 sample, initial states: backcasting)
Row	method	coverage (nominal level: 0.8)	MCE	scaled pinball (median)	coverage (nominal level: 0.8, level terciles: low)	coverage (nominal level: 0.8, level terciles: mid)	coverage (nominal level: 0.8, level terciles: high)	range of the 80% coverage over the level terciles	share of series for which dlnorm is selected
r1	auto.adam error selection	0.7461	0.0224	0.7514	0.7865	0.7425	0.7092	0.0773	0.6702
r2	AutoMSARIMA	0.6923	0.0352	0.6872	0.8193	0.7098	0.5477	0.2715

Calibration curves of the 99 quantiles on the fresh M4 sample: auto.msarima and auto.adam with error selection, overall, by level tercile and by the distributi… — A6 Calibration curves of the 99 quantiles on the fresh M4 sample: auto.msarima and auto.adam with error selection, overall, by level tercile and by the distribution auto.adam chose.

E13 · quant · benchmark · cited by C8, C10

Coverage by level terciles on the 5,315 benchmark series for AutoMSARIMA with backcasting (refitted) and for the other ARIMA implementations (saved forecasts of the smooth benchmark of 2026-11-05); and coverage of ETS and ARIMA on the series where the selected ETS(ANN) of ETS forced and selected and the selected ARIMA(0,1,1) give the same forecast and width (exploratory run RN15).

Setup: studied AutoMSARIMA, other ARIMA implementations, ETS forced and selected · cases benchmark · procedure level terciles · lens coverage, level ratio

E13.T1 · 80% coverage by level tercile of AutoMSARIMA and the other ARIMA implementations on the benchmark (data: benchmark)
Row	method	coverage (nominal level: 0.8, level terciles: low)	coverage (nominal level: 0.8, level terciles: mid)	coverage (nominal level: 0.8, level terciles: high)
r1	AutoMSARIMA	0.7862	0.6677	0.5897
r2	statsforecast AutoARIMA	0.7891	0.6672	0.5893
r3	pmdarima auto_arima	0.7702	0.6354	0.5825
r4	skforecast Arima	0.7809	0.6578	0.5854

E13.T2 · Series of the benchmark where the selected ETS(ANN) and the selected ARIMA(0,1,1) give the same forecast and width (data: benchmark)
Row	method	series	coverage (nominal level: 0.8)
r1	both	114
r2	ETS		0.7336
r3	ARIMA		0.7336

E14 · quant · benchmark · cited by C11

ETS forced and selected and ARIMA(0,1,1) forced on all 5,315 benchmark series and the selected ETS and AutoMSARIMA models, backcasting; cov80 per method (exploratory run RN14 on the forecasts of RN13).

Setup: studied ETS forced and selected, AutoMSARIMA · cases benchmark · lens coverage

E14.T1 · 80% coverage of forced and selected models on all 5,315 benchmark series, backcasting (data: benchmark)
Row	method	coverage (nominal level: 0.8)
r1	ARIMA(0,1,1)	0.6951
r2	ETS(ANN)	0.6681
r3	ETS(MNN)	0.7635
r4	ETS, selected	0.7492
r5	ARIMA, selected	0.6812

E15 · quant · benchmark · cited by C19

AutoMSARIMA and the log-normal ARIMA with the same selected orders on the 5,315 benchmark series, backcasting; cov80, MCE, scaled pinball and cov80 per level terciles (exploratory run RN38).

Setup: studied AutoMSARIMA, log-normal ARIMA · cases benchmark · procedure level terciles · lens coverage, MCE, scaled pinball, level ratio

E15.T1 · AutoMSARIMA and the log-normal ARIMA with the same orders on the benchmark, backcasting (data: benchmark, initial states: backcasting)
Row	method	coverage (nominal level: 0.8)	MCE	scaled pinball (median)	coverage (nominal level: 0.8, level terciles: low)	coverage (nominal level: 0.8, level terciles: mid)	coverage (nominal level: 0.8, level terciles: high)	scaled pinball (mean)
r1	AutoMSARIMA	0.6812	0.0402	0.4959	0.7862	0.6677	0.5897	0.8991
r2	log-normal ARIMA	0.7563	0.0124	0.5507	0.7493	0.7514	0.7683	1.0916

E16 · quant · benchmark · cited by C20

R auto.msarima (AutoMSARIMA) and auto.adam error selection on the 5,315 benchmark series, backcasting; 99 quantiles per series; MCE, cov80 by level terciles, scaled pinball and the share choosing dlnorm (exploratory run RN45). Run in R because Python's AutoADAM removes dlnorm from the candidate distributions of a pure ARIMA whatever the user passes.

Setup: studied auto.adam error selection, AutoMSARIMA · cases benchmark · procedure level terciles · lens coverage, MCE, scaled pinball, level ratio

E16.T1 · R auto.msarima and auto.adam error selection on the benchmark, backcasting (data: benchmark, initial states: backcasting)
Row	method	series fitted	series that failed	coverage (nominal level: 0.8)	share of holdout observations at or below the 0.1 quantile	share of holdout observations at or below the 0.5 quantile	share of holdout observations at or below the 0.9 quantile	MCE	scaled pinball (mean)	scaled pinball (median)	coverage (nominal level: 0.8, level terciles: low)	coverage (nominal level: 0.8, level terciles: mid)	coverage (nominal level: 0.8, level terciles: high)	range of the 80% coverage over the level terciles	share of series for which dlnorm is selected
r1	auto.adam error selection	5315	0	0.7291	0.1421	0.5024	0.8712	0.0187	0.9225	0.5137	0.7539	0.7102	0.7233	0.0436	0.642
r2	AutoMSARIMA	5315	0	0.6812	0.1436	0.4703	0.8248	0.0402	0.8991	0.4959	0.7862	0.6674	0.59	0.1962	0

E16.T2 · The same by the error distribution selected (data: benchmark, initial states: backcasting)
Row	method	selected distribution	coverage (nominal level: 0.8)	MCE
r1	auto.adam error selection	dlnorm	0.7503	0.0272
r2	auto.adam error selection	dnorm	0.6912	0.0442
r3	AutoMSARIMA	dnorm	0.6812	0.0402

Calibration curves of the 99 quantiles on the 5,315 benchmark series: auto.msarima, auto.adam with error selection and the selected ETS, overall, by level terc… — A5 Calibration curves of the 99 quantiles on the 5,315 benchmark series: auto.msarima, auto.adam with error selection and the selected ETS, overall, by level tercile and by the distribution auto.adam chose.

E17 · quant · benchmark · cited by C9

AutoMSARIMA forecasts of the 5,315 benchmark series with smooth 1.0.8 and 1.3.0 decomposed in stages: A, 1.0.8; D, 1.3.0 with the parameters of A; B, 1.3.0 with the orders of A; C, 1.3.0. Width ratios are medians over series of the 80% interval width of the later stage over the earlier one (exploratory run RN20 on runs RN16 to RN19).

Setup: studied AutoMSARIMA · cases benchmark · lens coverage · context smooth 1.0.8 and 1.3.0

E17.T1 · Stages of the change from smooth 1.0.8 to 1.3.0 on the benchmark (data: benchmark)
Row	initial states	stage	median ratio of the 80% interval width of the later stage to the earlier	change of 80% coverage	share of selected models with drift
r1	backcasting	A->C	1	-0.0189
r2	optimal	A->C	1
r3	backcasting	A->D	0.9871
r4	backcasting	C			0.2476
r5	backcasting	A			0.3087

E18 · quant · benchmark · cited by C12

500 benchmark series drawn with seed 41: R msarima with the orders Python selected, and R auto.msarima against Python AutoMSARIMA, backcasting; ratio of the 80% interval widths and coverage (exploratory run RN37).

Setup: studied AutoMSARIMA · cases benchmark · lens coverage

E18.T1 · R against Python AutoMSARIMA on 500 benchmark series drawn with seed 41, backcasting
Row	median ratio of the R to the Python 80% interval width	share of series where R and Python select the same model	coverage (nominal level: 0.8, implementation: R)	coverage (nominal level: 0.8, implementation: Python)
r1	1	1	0.6766	0.6766

E19 · quant · other · cited by C21

Coverage the known-model simulation predicts at the lengths of the 5,315 benchmark series: the pooled cov80 of each case of cases T, E, S, W interpolated on log T at each series length (lengths below 15 and above 300 at the nearest size) and averaged over the series (exploratory run RN40 on runs RN8, RN11, RN39).

Setup: studied AutoMSARIMA · cases known-model simulation, benchmark · procedure cases T, E, S, W · lens coverage

E19.T1 · 80% coverage the known-model simulation predicts at the lengths of the benchmark series (data: benchmark)
Row	cases T, E, S, W	coverage (nominal level: 0.8)
r1	T	0.8012
r2	E	0.725
r3	S	0.7169

E20 · quant · benchmark · cited by C22, C23

MCE of each calibration curve of the 99 quantiles drawn from recorded outputs: the known-model simulation (runs RN8, RN11, RN39), smooth 1.0.8 and 1.3.0 (RN16, RN19, RN43, RN44), the benchmark (RN45 and the saved ETS benchmark) and the fresh M4 sample (RN48), overall and by level terciles (exploratory run RN49).

Setup: studied AutoMSARIMA, auto.adam error selection, ETS forced and selected · cases known-model simulation, benchmark, fresh M4 sample · procedure level terciles · lens MCE

E20.T1 · MCE of each curve of figure A3
Row	panel	line	MCE
r1	T=15	true parameters	0.0144
r2	T=15	estimated parameters	0.0653
r3	T=15	selected orders	0.0579
r4	T=25	true parameters	0.0064
r5	T=25	estimated parameters	0.0341
r6	T=25	selected orders	0.0366
r7	T=50	true parameters	0.001
r8	T=50	estimated parameters	0.0196
r9	T=50	selected orders	0.0235
r10	T=100	true parameters	0.0023
r11	T=100	estimated parameters	0.01
r12	T=100	selected orders	0.0139
r13	T=300	true parameters	0.0007
r14	T=300	estimated parameters	0.0026
r15	T=300	selected orders	0.0046

E20.T2 · MCE of each curve of figure A4
Row	panel	line	MCE
r1	benchmark, backcasting	1.0.8	0.0326
r2	benchmark, backcasting	1.3.0	0.0402
r3	benchmark, optimal	1.0.8	0.0331
r4	benchmark, optimal	1.3.0	0.039
r5	simulation T=100, true parameters	1.0.8	0.0123
r6	simulation T=100, true parameters	1.3.0	0.0023
r7	simulation T=100, estimated	1.0.8	0.012
r8	simulation T=100, estimated	1.3.0	0.01
r9	simulation T=100, selected	1.0.8	0.0172
r10	simulation T=100, selected	1.3.0	0.0139

E20.T3 · MCE of each curve of figure A5
Row	panel	line	MCE
r1	all series	AutoMSARIMA	0.0402
r2	all series	auto.adam error selection	0.0187
r3	all series	ETS (AutoADAM ZXZ, saved)	0.0442
r4	low level tercile	AutoMSARIMA	0.0281
r5	low level tercile	auto.adam error selection	0.0334
r6	mid level tercile	AutoMSARIMA	0.0505
r7	mid level tercile	auto.adam error selection	0.0352
r8	high level tercile	AutoMSARIMA	0.0623
r9	high level tercile	auto.adam error selection	0.0655
r10	auto.adam by selected distribution	chose dnorm	0.0442
r11	auto.adam by selected distribution	chose dlnorm	0.0272

E20.T4 · MCE of each curve of figure A6
Row	panel	line	MCE
r1	all series	AutoMSARIMA	0.0352
r2	all series	auto.adam error selection	0.0224
r3	low level tercile	AutoMSARIMA	0.0348
r4	low level tercile	auto.adam error selection	0.041
r5	mid level tercile	AutoMSARIMA	0.0412
r6	mid level tercile	auto.adam error selection	0.02
r7	high level tercile	AutoMSARIMA	0.0757
r8	high level tercile	auto.adam error selection	0.1052
r9	auto.adam by selected distribution	chose dnorm	0.0209
r10	auto.adam by selected distribution	chose dlnorm	0.0351

W1 · argument · abductive · premises C10, C11

Where ETS and ARIMA reduce to the same model, their intervals coincide and cover the same share (C17), so the interval computations agree; the additive ETS(ANN) forced on every series covers no more than ARIMA(0,1,1), while the multiplicative ETS(MNN) covers 0.07 to 0.10 more (C22). The difference between the pools is therefore the multiplicative-error models, which AutoMSARIMA does not have.

Limitations

L1 · sampling · could bias magnitude · affects C16 · mitigation: none

The 12 wrong models of case W were chosen by the authors, not drawn from the mistakes AutoMSARIMA makes, so the size of the coverage loss they produce depends on that choice.

L2 · other · reduces precision · affects C2, C3, C4 · mitigation: none

The lab record flags plans PL11, PL12 and PL13 with code changed after the plan; the only commit between those plans and their runs updated the claimset skills under .claude/skills, not the analysis scripts.

L3 · measurement · reduces transferability · affects C2 · mitigation: none

ETS(MNN), the comparator whose coverage stays flat over the level terciles on M4, is badly centred there (q50 of 0.372) and has an MCE of 0.0849, so its flat cov80 does not make it well calibrated.

L4 · computational · could bias magnitude · affects C4 · mitigation: none

smooth 1.0.8 ran from a separate worktree at tag v1.0.8-py with its own Python environment, so the versions also differ in the environment around the package.

L5 · computational · reduces transferability · affects C5, C20 · mitigation: none

The auto.adam comparison ran in R, because Python's AutoADAM removes dlnorm from the candidate distributions of a pure ARIMA whatever the user passes, while R's auto.adam removes it only from its default list.

L6 · analytic flexibility · could bias magnitude · affects C5 · mitigation: none

The thresholds of plan PL15 were fixed after the exploratory benchmark run of auto.adam (RN45) was seen, on M4 series that run did not use.

L7 · sample size · reduces precision · affects C10 · mitigation: none

The series where the selected ETS(ANN) and ARIMA(0,1,1) give the same forecast and width are 114 of 5,315.

L8 · model specification · could bias magnitude · affects C21 · mitigation: none

The prediction of coverage at the benchmark lengths uses Gaussian processes and interpolates the simulated coverage over log length, with lengths outside 15 to 300 at the nearest simulated size.

Open questions

O1 · from C5, C22

Why do the quantiles of auto.adam with error selection lie too high, by about 0.08 to 0.13 of coverage between the 0.2 and 0.6 quantiles, on the series of the highest level tercile, and would a model of the location of grown series correct it?

O2 · from C15, C17

Why do the parameter draws of reforecast() give explosive or several times too wide intervals for ARIMA models with a constant or AR terms, differently in R and Python?

O3 · from C3, C19, C20

Would AutoMSARIMA calibrate better with error selection than with a fixed log-normal error once the location error of grown series is addressed, given that the fixed log-normal error costs 11% of median pinball on the benchmark?

O4 · from C4, C23

Which of the changes between smooth 1.0.8 and 1.3.0 that alter model selection (fewer drift models chosen with backcasting) makes the benchmark calibration of AutoMSARIMA worse while its calibration on known models improves?

References

Makridakis et al. (1982) Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E., & Winkler, R. (1982). The accuracy of extrapolation (time series) methods: Results of a forecasting competition. Journal of Forecasting, 1(2), 111–153. doi:10.1002/for.3980010202 Used by: D18
Makridakis and Hibon (2000) Makridakis, S., & Hibon, M. (2000). The M3-Competition: results, conclusions and implications. International Journal of Forecasting, 16(4), 451–476. doi:10.1016/S0169-2070(00)00057-1 Used by: D19
Athanasopoulos et al. (2011) Athanasopoulos, G., Hyndman, R. J., Song, H., & Wu, D. C. (2011). The tourism forecasting competition. International Journal of Forecasting, 27(3), 822–844. doi:10.1016/j.ijforecast.2010.04.009 Used by: D20
Makridakis et al. (2020) Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2020). The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1), 54–74. doi:10.1016/j.ijforecast.2019.04.014 Used by: D6, D9
Chatfield (1993) Chatfield, C. (1993). Calculating Interval Forecasts. Journal of Business & Economic Statistics, 11(2), 121–135. doi:10.1080/07350015.1993.10509938 Used by: Context
Svetunkov (2023) Svetunkov, I. (2023). Forecasting and Analytics with the Augmented Dynamic Adaptive Model (ADAM). Chapman and Hall/CRC. doi:10.1201/9781003452652 Used by: Context, D1, D8, D10, D11, D15

Artefacts

Score

Reliability of the paper: 60, Medium

Scores are computed by rules from the claimset file (EntryLib rubric 0.2): they check the record, not the science, and may change as the rubric and reviews develop. The rubric. Every issue as text, for an agent preparing the next version.

The paper scores as its lowest principal claim, C6. Open problems in the principal claims and what they rest on, each counted once, with the most it costs a claim:

−30 Support Rests on argument only. In C6. To remove it: Support the claim with evidence the claimset records, or have its proof checked.
−12 Open problems Limitation L4 (could bias magnitude, not mitigated): smooth 1.0.8 ran from a separate worktree at tag v1.0.8-py with its own Python environment, so the versions also differ in the environment around the package. In C4. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
−12 Open problems Limitation L6 (could bias magnitude, not mitigated): The thresholds of plan PL15 were fixed after the exploratory benchmark run of auto.adam (RN45) was seen, on M4 series that run did not use. In C5. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
−5 Context A principal claim related to no earlier work. In C3, C4, C5, C6. To remove it: Relate the claim to the earlier work it extends, addresses or departs from.
−5 Plan The lab record flags the test: code changed after plan. In C2, C3, C4. To remove it: Rerun the test as planned, or explain the flag in a limitation.
−3 Plan The plan's time is anchored by a git push only. In C1, C2, C3, C4, C5, C7. To remove it: Anchor plans with the platform, a timestamp authority (RFC 3161) or a registry.
−3 Open problems Assumption C24 is not checked: The coverage of a set of series is well summarised by first averaging the holdout observations within each series and then averaging over series, all horizons together. In C1. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
−5 Open problems Limitation L2 (reduces precision, not mitigated): The lab record flags plans PL11, PL12 and PL13 with code changed after the plan; the only commit between those plans and their runs updated the claimset skills under .claude/skills, not the analysis scripts. In C2, C3, C4. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
−1 Open problems Limitation L3 (reduces transferability, not mitigated): ETS(MNN), the comparator whose coverage stays flat over the level terciles on M4, is badly centred there (q50 of 0.372) and has an MCE of 0.0849, so its flat cov80 does not make it well calibrated. In C2. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
−2 Open problems Limitation L5 (reduces transferability, not mitigated): The auto.adam comparison ran in R, because Python's AutoADAM removes dlnorm from the candidate distributions of a pure ARIMA whatever the user passes, while R's auto.adam removes it only from its default list. In C5. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
−5 Open problems Limitation L7 (reduces precision, not mitigated): The series where the selected ETS(ANN) and ARIMA(0,1,1) give the same forecast and width are 114 of 5,315. In C10. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
−3 Open problems Assumption C25 is not checked: The ratio of the mean of the last third of a series to its overall mean indexes how much the scale of its errors at the forecast origin exceeds the average scale over the sample. In C2, C8. To remove it: Test the assumption, or analyse how sensitive the claim is to it.