CLAIMSET Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4
licence CC-BY-4.0; contributions C1, C2, C3, C4, C5, C6; highlights C1, C2, C3, C5
declarations funding: none | competing interests: The guarantor is the author and maintainer of the smooth package that the study evaluates.
exploration {"lab_record": "A7", "questions": 5, "hypotheses": {"total": 11, "supported": 6, "not_supported": 3, "inconclusive": 2, "revised": 0, "abandoned": 0, "open": 0, "from_exploration": 8}, "tests": {"total": 12, "supported": 6, "not_supported": 3, "inconclusive": 3}, "runs": {"total": 49, "confirmatory": 13, "exploratory": 36, "failed": 3}}

DEFINITIONS
D1 [method] MSARIMA analytic interval: The quantiles of `predict(interval="prediction")` of a pure ARIMA in smooth 1.3.0, which dispatches to the approximate interval. The ARIMA is the state space model $y_t = w^\top v_{t-l} + \epsilon_t$, $v_t = F v_{t-l} + g\epsilon_t$, $\epsilon_t \sim N(0, \sigma^2)$. The quantile at level $\tau$ and horizon $h$ is $q_{\tau,h} = \hat y_{T+h|T} + \Phi^{-1}(\tau)\sqrt{V_h}$ with $V_h = \hat\sigma^2 \frac{T}{T-k} c_h$, where $\hat\sigma^2 = \frac{1}{T}\sum_{t=1}^{T} e_t^2$, $k$ the number of estimated parameters and $c_h$ the $h$-th diagonal element of the analytic covariance of the state space model, $c_h = 1 + \sum_{j=1}^{h-1} (w^\top F^{j-1} g)^2$ for a model with lag 1. [source B6]
D2 [data] known-model simulation: Gaussian ARIMA series with $\sigma = 10$ generated from 11 data generating processes (white noise with a constant, random walk with and without drift, AR(1), MA(1), ARIMA(0,1,1), ARIMA(1,1,0), ARIMA(1,1,1), ARIMA(0,2,2), seasonal SARIMA(0,1,1)(0,1,1)$_{12}$ and SARIMA(1,0,0)(1,0,0)$_{12}$) plus an IMA(1,1) with Student-t errors on 4 degrees of freedom; in-sample lengths $T$ of 15, 25, 50, 100 and 300, holdout of 18; 2,000 replicates per process and length with seed 20261105 unless stated.
D3 [design] cases T, E, S, W: What the fitted AutoMSARIMA or MSARIMA knows about the data generating process of the [[D2:known-model simulation]]: T, true orders and true AR, MA and constant parameters with initial states and scale estimated; E, true orders with estimated parameters; S, orders selected by AutoMSARIMA; W, a fixed wrong model for each process. Each with backcasting or optimal initial states.
D4 [variable] coverage: For a quantile $q_\tau$, the share of holdout observations at or below it, $c_\tau = \frac{1}{h}\sum_{j=1}^{h} 1(y_{T+j} \le q_{\tau,j})$, averaged first within a series and then over series. cov80 is $c_{0.9} - c_{0.1}$, the coverage of the central 80% interval; q50 is $c_{0.5}$.
D5 [data] benchmark: The 5,315 series of M1 (1,001), M3 (3,003) and Tourism (1,311) from fcompdata 0.1.4, with their own holdouts and seasonal periods; lags $(1, m)$ for seasonal series. [members D18, D19, D20]
D6 [data] M4 sample: 1,500 series each of the yearly, quarterly and monthly M4 series, drawn with seed 20261105 without replacement, with the M4 holdouts. [source B4]
D7 [design] level terciles: Series split into terciles of the [[D17:level ratio]] across the series of a data set; cov80 reported for each tercile, and the spread as the largest minus the smallest of the three.
D8 [method] auto.adam error selection: R `auto.adam(y, model="NNN", lags, distribution=c("dnorm","dlnorm"), initial="backcasting")` of smooth 4.6: the ARIMA orders and constant are selected by AICc for each distribution with the maxima of auto.msarima (AR (3,3), I (2,1), MA (3,3)), then the distribution with the lower AICc is kept; with dnorm the model is the [[D10:AutoMSARIMA]] model, with dlnorm it is the [[D11:log-normal ARIMA]]. Quantiles from `forecast(interval="prediction")`. [source B6]
D9 [data] fresh M4 sample: 1,500 series each of the yearly, quarterly and monthly M4 series, drawn with seed 20261107 from the series not in the [[D6:M4 sample]], so disjoint from it. [source B4]
D10 [method] AutoMSARIMA: `AutoMSARIMA(lags, initial)` in Python and `auto.msarima()` in R of smooth 1.3.0 / 4.6: ARIMA orders up to AR (3,3), I (2,1), MA (3,3) and a constant selected by AICc, normal errors, the [[D1:MSARIMA analytic interval]] for the quantiles. Backcasting initial states unless stated. [source B6]
D11 [method] log-normal ARIMA: ADAM with `model="NNN"`, an ARIMA and `distribution="dlnorm"`: the ARIMA applies to the logarithms, $\log y_t = w^\top v_{t-l} + \log(1+\epsilon_t)$, $v_t = F v_{t-l} + g \log(1+\epsilon_t)$, $1+\epsilon_t \sim \mathrm{logN}(-\sigma^2/2, \sigma^2)$, so the scale of the error is proportional to the level. In the comparisons with fixed orders, the orders and constant are those [[D10:AutoMSARIMA]] selects on the same series. [source B6]
D12 [method] reforecast: `reforecast()` of smooth: draws the parameters 300 times from a normal distribution with the estimated covariance of the parameters, reapplies the model with each draw and simulates the predictive distribution, so that the quantiles include the uncertainty of the parameters.
D13 [variable] MCE: Mean calibration error over the 99 quantile levels $\tau = 0.01, \dots, 0.99$: $\frac{1}{99}\sum_\tau |\bar c_\tau - \tau|$, with $\bar c_\tau$ the [[D4:coverage]] of $q_\tau$.
D14 [variable] scaled pinball: Pinball loss averaged over the 99 quantile levels and the horizons, divided by the mean absolute first difference of the in-sample data for real series; in the simulation, by $\sigma$, and relpin is its ratio to the pinball of the true predictive distribution.
D15 [method] ETS forced and selected: ETS of smooth with backcasting: AutoADAM-style selection among ETS models (`ES(model="ZXZ")`, or ZXN for non-seasonal series), or ETS(ANN) or ETS(MNN) fitted to every series. [source B6]
D16 [method] other ARIMA implementations: statsforecast AutoARIMA, pmdarima auto_arima and skforecast Arima, as run in the notebook of the smooth benchmark of 2026-11-05, whose saved forecasts were read.
D17 [variable] level ratio: Mean of the last third of the in-sample data of a series divided by the mean of all its in-sample data.
D18 [data] M1: The 1,001 series of the M1 competition with their holdouts. [source B1]
D19 [data] M3: The 3,003 series of the M3 competition with their holdouts. [source B2]
D20 [data] Tourism: The 1,311 series of the tourism forecasting competition with their holdouts. [source B3]
D21 [setting] smooth 1.0.8 and 1.3.0: Python smooth at tag v1.0.8-py (a separate worktree with its own environment) and at 1.3.0 (commit 634657d8 of openforecast-org/smooth).

CLAIMS
C1 [descriptive; principal; support: simulation E1 (E1.T1.r3, E1.T1.r2, E1.T1.r1) artefacts A3, A8; assumes C24; uses C7, C12; strength high (author); verifiability open; planning confirmatory; outcome supported]
  With the true orders and true parameters, the [[D1:MSARIMA analytic interval]] is calibrated over the 99 quantiles in the [[D2:known-model simulation]] at $T$ of 100 and 300: [[D13:MCE]] lies between 0.0007 and 0.0024 and the ratio of the interval width to the theoretical width between 0.9965 and 1.0019, with backcasting and with optimal initial states.
  scope: data_type: simulated Gaussian ARIMA and SARIMA series | conditions: holdout of 18; smooth 1.3.0 (Python); case T of the cases T, E, S, W; T of 100 and 300
  addresses B6: The book derives the interval; this checks the implementation of smooth 1.3.0 against it.
  test: deterministic; sample 2000 replicates per cell, h=1..18, fixed seed 20261105, no interim analysis; supported if m_b100<0.01 and x_b100<0.02 and abs(w_b100-1)<0.02 and m_o100<0.01 and x_o100<0.02 and abs(w_o100-1)<0.02 and m_b300<0.01 and x_b300<0.02 and abs(w_b300-1)<0.02 and m_o300<0.01 and x_o300<0.02 and abs(w_o300-1)<0.02; not supported if m_b100>0.02 or x_b100>0.04 or abs(w_b100-1)>0.04 or m_o100>0.02 or x_o100>0.04 or abs(w_o100-1)>0.04 or m_b300>0.02 or x_b300>0.04 or abs(w_b300-1)>0.04 or m_o300>0.02 or x_o300>0.04 or abs(w_o300-1)>0.04; verdict computed
  plan: lab plan PL4; planned 2026-11-05T11:37:01Z; anchor git_remote
C2 [descriptive; principal; support: empirical_quant E8 (E8.T1.r1, E8.T1.r2); assumes C25; uses C8; limitations L2, L3; strength moderate (author); verifiability open; planning confirmatory; outcome supported]
  On the [[D6:M4 sample]], the 80% [[D4:coverage]] of [[D10:AutoMSARIMA]] falls with the [[D17:level ratio]]: 0.813, 0.700 and 0.554 from the lowest to the highest of the [[D7:level terciles]], while ETS(MNN) of [[D15:ETS forced and selected]], whose error scale follows the level, covers 0.764, 0.778 and 0.814.
  scope: data_type: M4 yearly, quarterly and monthly series | conditions: backcasting; smooth 1.3.0 (Python)
  addresses B5: Chatfield discusses why intervals are too narrow in general; this locates the shortfall of one implementation on series whose level has grown.
  test: deterministic; sample 4500 M4 series (1500 yearly, quarterly, monthly), seed 20261105, terciles of the recent level over the mean level from the first method; supported if fa >= 0.10 and fm <= 0.05; not supported if fa < 0.05; verdict computed; flags code_changed_after_plan
  plan: lab plan PL12; planned 2026-11-05T18:40:25Z; anchor git_remote
C3 [descriptive; principal; support: empirical_quant E9 (E9.T1.r2, E9.T1.r1); uses C2; limitations L2; strength moderate (author); verifiability open; planning confirmatory; outcome supported]
  On the [[D6:M4 sample]], the [[D11:log-normal ARIMA]] with the orders [[D10:AutoMSARIMA]] selects removes the fall of 80% [[D4:coverage]] over the [[D7:level terciles]] (0.774, 0.783 and 0.752, a spread of 0.031) and lowers [[D13:MCE]] from 0.0386 to 0.0177.
  scope: data_type: M4 yearly, quarterly and monthly series | conditions: backcasting; orders fixed at those AutoMSARIMA selects
  test: deterministic; sample 4500 M4 series as in the plan of H8, same seed; supported if sd <= 0.05 and md <= 0.5*ma; not supported if sd > 0.10 or md >= ma; verdict computed; flags code_changed_after_plan
  plan: lab plan PL13; planned 2026-11-05T18:40:25Z; anchor git_remote
C4 [descriptive; principal; support: simulation E10 (E10.T1.r1, E10.T1.r5, E10.T1.r3, E10.T1.r7, E10.T1.r2, E10.T1.r6) artefacts A4, A8; uses C9; limitations L2, L4; strength high (author); verifiability open; planning confirmatory; outcome supported]
  On known models, the [[D10:AutoMSARIMA]] interval of smooth 1.3.0 is calibrated no worse than that of 1.0.8 ([[D21:smooth 1.0.8 and 1.3.0]]): at $T = 100$ in the [[D2:known-model simulation]] its [[D13:MCE]] is 0.0100 against 0.0120 with estimated parameters, 0.0139 against 0.0172 with selected orders and 0.0085 against 0.0246 with optimal initial states.
  scope: data_type: simulated Gaussian ARIMA and SARIMA series | conditions: holdout of 18; T = 100; cases E and S
  test: deterministic; sample 2000 replicates per cell, 11 DGPs, T=50,100,300, seed 20261105 in both versions; run 1 smooth 1.0.8 (v1.0.8-py), run 2 smooth 1.3.0; supported if eb130 <= eb108+0.001 and sb130 <= sb108+0.001 and eo130 <= eo108+0.001; not supported if eb130 > eb108+0.005 or sb130 > sb108+0.005 or eo130 > eo108+0.005; verdict computed; flags code_changed_after_plan
  plan: lab plan PL11; planned 2026-11-05T18:40:15Z; anchor git_remote
C5 [descriptive; principal; support: empirical_quant E12 (E12.T1.r1, E12.T1.r2) artefacts A6; limitations L5, L6; strength moderate (author); verifiability open; planning confirmatory; outcome inconclusive]
  On the [[D9:fresh M4 sample]], [[D8:auto.adam error selection]] lowers [[D13:MCE]] from 0.0352 to 0.0224 and the spread of 80% [[D4:coverage]] over the [[D7:level terciles]] from 0.272 to 0.077 against R auto.msarima ([[D10:AutoMSARIMA]]), short of the planned reduction of MCE to 0.6 times that of auto.msarima, so the planned test is inconclusive.
  scope: data_type: M4 yearly, quarterly and monthly series | conditions: backcasting; R smooth 4.6
  test: deterministic; sample 4500 fresh M4 series, one run; supported if ms <= 0.6*ma and ss <= 0.08; not supported if ms >= ma or ss >= sa; verdict computed
  plan: lab plan PL15; planned 2026-11-09T17:07:13Z; anchor git_remote
C6 [interpretive; principal; support: argument W1; uses C10, C11; strength moderate (author); verifiability open; planning exploratory]
  The wider intervals of ETS than of [[D10:AutoMSARIMA]] on the [[D5:benchmark]] come from the multiplicative-error models in the ETS pool of [[D15:ETS forced and selected]], not from how either package computes the interval.
  scope: data_type: M1, M3 and Tourism series | conditions: backcasting
C7 [descriptive; supporting; support: simulation E3 (E3.T1.r1, E3.T1.r2, E3.T1.r3, E3.T1.r4, E3.T1.r5, E3.T1.r6) artefacts A8; strength high (author); verifiability open; planning confirmatory; outcome supported]
  With estimated parameters in the [[D2:known-model simulation]], the [[D1:MSARIMA analytic interval]] and the simulated interval of the same fit have the same width and [[D13:MCE]] at $T$ of 50, 100 and 300 (widths 0.957 and 0.957, 0.981 and 0.980, 0.997 and 0.996; MCE 0.0196 and 0.0197, 0.0100 and 0.0100, 0.0026 and 0.0027).
  scope: data_type: simulated Gaussian ARIMA and SARIMA series | conditions: holdout of 18; smooth 1.3.0 (Python); case E; backcasting
  addresses B6: Checks that the analytic variance of the implementation matches simulation from the same model.
  test: deterministic; sample 2000 replicates per cell, h=1..18, fixed seed 20261105, no interim analysis; supported if abs(ws50/wa50-1)<0.01 and abs(ms50-ma50)<0.003 and abs(ws100/wa100-1)<0.01 and abs(ms100-ma100)<0.003 and abs(ws300/wa300-1)<0.01 and abs(ms300-ma300)<0.003; not supported if abs(ws50/wa50-1)>0.03 or abs(ms50-ma50)>0.01 or abs(ws100/wa100-1)>0.03 or abs(ms100-ma100)>0.01 or abs(ws300/wa300-1)>0.03 or abs(ms300-ma300)>0.01; verdict computed
  plan: lab plan PL6; planned 2026-11-05T11:37:02Z; anchor git_remote
C8 [descriptive; supporting; support: empirical_quant E13 (E13.T1.r1, E13.T1.r2, E13.T1.r3, E13.T1.r4); assumes C25; strength moderate (author); verifiability open; planning exploratory]
  On the 5,315 [[D5:benchmark]] series, the fall of 80% [[D4:coverage]] over the [[D7:level terciles]] is shared by [[D10:AutoMSARIMA]] (0.786, 0.668, 0.590) and the [[D16:other ARIMA implementations]]: statsforecast AutoARIMA (0.789, 0.667, 0.589), pmdarima (0.770, 0.635, 0.582) and skforecast (0.781, 0.658, 0.585).
  scope: data_type: M1, M3 and Tourism series | conditions: backcasting for smooth
C9 [descriptive; supporting; support: empirical_quant E17 (E17.T1.r1, E17.T1.r2); strength moderate (author); verifiability open; planning exploratory]
  The width of the [[D10:AutoMSARIMA]] interval on the [[D5:benchmark]] did not change between [[D21:smooth 1.0.8 and 1.3.0]]: the median ratio of the 80% widths of 1.3.0 over 1.0.8 is 1.000 with backcasting and with optimal initial states.
  scope: data_type: M1, M3 and Tourism series
C10 [descriptive; supporting; support: empirical_quant E13 (E13.T2.r1, E13.T2.r2, E13.T2.r3); limitations L7; strength moderate (author); verifiability open; planning exploratory]
  On the 114 [[D5:benchmark]] series where the selected ETS(ANN) of [[D15:ETS forced and selected]] and the ARIMA(0,1,1) selected by [[D10:AutoMSARIMA]] give the same forecast and width, the two have the same 80% [[D4:coverage]], 0.734.
  scope: data_type: M1, M3 and Tourism series | conditions: backcasting
C11 [descriptive; supporting; support: empirical_quant E14 (E14.T1.r3, E14.T1.r2, E14.T1.r1); strength moderate (author); verifiability open; planning exploratory]
  Forced on all 5,315 [[D5:benchmark]] series, ETS(MNN) of [[D15:ETS forced and selected]] has an 80% [[D4:coverage]] of 0.764, against 0.668 for ETS(ANN) and 0.695 for ARIMA(0,1,1).
  scope: data_type: M1, M3 and Tourism series | conditions: backcasting
C12 [descriptive; supporting; not_applicable; support: empirical_quant E18 (E18.T1.r1); strength moderate (author); verifiability open; planning exploratory]
  R and Python give the same [[D10:AutoMSARIMA]] interval on 500 [[D5:benchmark]] series: the median ratio of the 80% widths is 1.000, the same model is selected on every series and both have an 80% [[D4:coverage]] of 0.677.
  scope: data_type: 500 M1, M3 and Tourism series | conditions: backcasting
C24 [descriptive; supporting; assumption (methodological, unchecked); verifiability not_applicable]
  The coverage of a set of series is well summarised by first averaging the holdout observations within each series and then averaging over series, all horizons together.
C25 [descriptive; supporting; assumption (data, unchecked); verifiability not_applicable]
  The ratio of the mean of the last third of a series to its overall mean indexes how much the scale of its errors at the forecast origin exceeds the average scale over the sample.
C13 [descriptive; secondary; extension; support: simulation E2 (E2.T1.r1) artefacts A8; strength moderate (author); verifiability open; planning confirmatory; outcome inconclusive]
  Whether selecting the orders with [[D10:AutoMSARIMA]] loses little [[D4:coverage]] in the [[D2:known-model simulation]] at $T = 100$ was not settled: pooled cov80 with selected orders was 0.756, between the threshold for little loss (0.77) and that for a large loss (0.74).
  scope: data_type: simulated Gaussian ARIMA and SARIMA series | conditions: holdout of 18; smooth 1.3.0 (Python); case S; T = 100; backcasting
  extends B5: Chatfield lists model selection among the reasons intervals are too narrow; this measures it for AutoMSARIMA.
  test: deterministic; sample 2000 replicates per cell, h=1..18, fixed seed 20261105, no interim analysis; supported if s >= 0.77; not supported if s < 0.74; verdict computed
  plan: lab plan PL5; planned 2026-11-05T11:37:02Z; anchor git_remote
C14 [descriptive; secondary; extension; support: simulation E4 (E4.T1.r2, E4.T1.r1) artefacts A8; strength high (author); verifiability open; planning confirmatory; outcome not_supported]
  Estimating the parameters does not cost 6 points of 80% [[D4:coverage]] of the [[D1:MSARIMA analytic interval]] at $T = 100$ in the [[D2:known-model simulation]]: with the true orders, pooled cov80 is 0.800 with true parameters and 0.770 with estimated parameters, a loss of 3 points.
  scope: data_type: simulated Gaussian ARIMA and SARIMA series | conditions: holdout of 18; smooth 1.3.0 (Python); cases T and E; T = 100; backcasting
  extends B5: Chatfield attributes narrow intervals partly to ignored parameter uncertainty; this sizes the effect at T = 100.
  test: deterministic; sample 2000 replicates per cell, h=1..18, fixed seed 20261105, no interim analysis; supported if t-e >= 0.06; not supported if t-e < 0.03; verdict computed
  plan: lab plan PL7; planned 2026-11-05T11:37:02Z; anchor git_remote
C15 [descriptive; secondary; support: simulation E5 (E5.T1.r1, E5.T1.r2, E5.T1.r3, E5.T1.r4) artefacts A8; strength high (author); verifiability open; planning confirmatory; outcome not_supported]
  [[D12:reforecast]] does not improve on the [[D1:MSARIMA analytic interval]] of `predict()` in the [[D2:known-model simulation]] at $T = 100$ with estimated parameters: with backcasting its [[D13:MCE]] is 0.0265 against 0.0100 and its relative [[D14:scaled pinball]] 1.958 against 1.043; with optimal initial states its MCE is lower (0.0053 against 0.0085) but its relative pinball higher (1.396 against 1.058).
  scope: data_type: simulated Gaussian ARIMA and SARIMA series | conditions: holdout of 18; smooth 1.3.0 (Python); case E; T = 100
  addresses B6: The book proposes reforecasting to include parameter uncertainty; this tests the implementation for ARIMA.
  test: deterministic; sample 2000 replicates per cell, h=1..18, fixed seed 20261105, no interim analysis; supported if rb<mb and sb<=pb and ro<mo and so<=po; not supported if rb>mb+0.002 or sb>pb*1.01 or ro>mo+0.002 or so>po*1.01; verdict computed
  plan: lab plan PL8; planned 2026-11-05T11:37:02Z; anchor git_remote
C16 [descriptive; secondary; support: simulation E6 (E6.T1.r1) artefacts A8; limitations L1; strength low (author); verifiability open; planning confirmatory; outcome supported]
  A fixed wrong model alone produces under-coverage of the size seen on the [[D5:benchmark]]: over 12 wrong models in the [[D2:known-model simulation]] at $T = 100$ the pooled 80% [[D4:coverage]] of the [[D1:MSARIMA analytic interval]] is 0.676 and its q50 0.428.
  scope: data_type: simulated Gaussian ARIMA and SARIMA series | conditions: holdout of 18; smooth 1.3.0 (Python); case W with 12 fixed wrong models; T = 100; backcasting
  test: deterministic; sample 2000 replicates per cell, h=1..18, fixed seed 20261105, no interim analysis; supported if w <= 0.72; not supported if w >= 0.76; verdict computed
  plan: lab plan PL9; planned 2026-11-05T11:37:03Z; anchor git_remote
C17 [descriptive; secondary; support: empirical_quant E7 (E7.T1.r1, E7.T1.r2); strength high (author); verifiability open; planning confirmatory; outcome not_supported]
  On the 5,315 [[D5:benchmark]] series, [[D12:reforecast]] does not close the gap of the 80% [[D4:coverage]] of [[D10:AutoMSARIMA]]: cov80 moves from 0.681 to 0.683 and its mean [[D14:scaled pinball]] is 7e191 against 0.899.
  scope: data_type: M1, M3 and Tourism series | conditions: backcasting; smooth 1.3.0 (Python)
  test: deterministic; sample All 5,315 series, seed per series, nsim=300; run only after the verdict of the plan testing reforecast() in simulation; supported if cr-cp >= 0.5*(0.8-cp) and pr <= 1.01*pp; not supported if cr-cp < 0.2*(0.8-cp) or pr > 1.05*pp; verdict computed
  plan: lab plan PL10; planned 2026-11-05T11:37:03Z; anchor git_remote
C18 [descriptive; secondary; not_applicable; support: empirical_quant E11; strength very_low (author); verifiability open; planning confirmatory; outcome inconclusive]
  The planned test of [[D8:auto.adam error selection]] against auto.msarima on the [[D9:fresh M4 sample]] under plan PL14 gave no result: its only run stopped on loading the M4 data, before any model was fitted.
  scope: data_type: M4 yearly, quarterly and monthly series
  test: deterministic; sample 4500 fresh M4 series, one run; supported if ms <= 0.6*ma and ss <= 0.08; not supported if ms >= ma or ss >= sa; verdict computed
  plan: lab plan PL14; planned 2026-11-07T14:02:07Z; anchor git_remote
C19 [descriptive; secondary; support: empirical_quant E15 (E15.T1.r1, E15.T1.r2); strength moderate (author); verifiability open; planning exploratory]
  On the 5,315 [[D5:benchmark]] series, the [[D11:log-normal ARIMA]] with the selected orders raises the 80% [[D4:coverage]] of [[D10:AutoMSARIMA]] from 0.681 to 0.756 and lowers [[D13:MCE]] from 0.0402 to 0.0124, but raises the median [[D14:scaled pinball]] from 0.496 to 0.551.
  scope: data_type: M1, M3 and Tourism series | conditions: backcasting; orders fixed at those AutoMSARIMA selects
C20 [descriptive; secondary; support: empirical_quant E16 (E16.T1.r1, E16.T1.r2) artefacts A5; limitations L5, L6; strength moderate (author); verifiability open; planning exploratory]
  On the 5,315 [[D5:benchmark]] series, [[D8:auto.adam error selection]] chooses dlnorm for 64% of the series, raises 80% [[D4:coverage]] from 0.681 to 0.729, lowers [[D13:MCE]] from 0.0402 to 0.0187 and raises the median [[D14:scaled pinball]] from 0.496 to 0.514 against R auto.msarima ([[D10:AutoMSARIMA]]).
  scope: data_type: M1, M3 and Tourism series | conditions: backcasting; R smooth 4.6
C21 [predictive; secondary; extension; support: empirical_quant E19 (E19.T1.r3, E19.T1.r1); limitations L8; strength low (author); verifiability open; planning exploratory]
  Estimation and selection alone, as measured in the [[D2:known-model simulation]], predict an 80% [[D4:coverage]] of 0.717 for [[D10:AutoMSARIMA]] at the lengths of the [[D5:benchmark]] series, against 0.801 with the true parameters.
  scope: data_type: simulated series at the lengths of the M1, M3 and Tourism series | conditions: backcasting; case S
  extends B5: Sizes the effect of estimation and selection on short series for one implementation.
C22 [descriptive; secondary; support: empirical_quant E20 (E20.T4.r3, E20.T4.r4, E20.T4.r5, E20.T4.r6, E20.T4.r7, E20.T4.r8) artefacts A6; strength moderate (author); verifiability open; planning exploratory]
  On the [[D9:fresh M4 sample]], [[D8:auto.adam error selection]] has a higher [[D13:MCE]] than R auto.msarima ([[D10:AutoMSARIMA]]) in the lowest and highest [[D7:level terciles]] (0.041 against 0.035, and 0.105 against 0.076) and a lower one only in the middle tercile (0.020 against 0.041), so its lower overall MCE comes in part from errors of opposite sign in different series.
  scope: data_type: M4 yearly, quarterly and monthly series | conditions: backcasting; R smooth 4.6
C23 [descriptive; secondary; support: empirical_quant E20 (E20.T2.r1, E20.T2.r2) artefacts A4; strength moderate (author); verifiability open; planning exploratory]
  On the 5,315 [[D5:benchmark]] series, the [[D13:MCE]] of [[D10:AutoMSARIMA]] is 0.0402 with smooth 1.3.0 against 0.0326 with 1.0.8 ([[D21:smooth 1.0.8 and 1.3.0]]) with backcasting.
  scope: data_type: M1, M3 and Tourism series | conditions: backcasting

EVIDENCE
E1 [simulation; runs 2000] Simulation of the [[D2:known-model simulation]]: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Case T of [[D3:cases T, E, S, W]] with backcasting and optimal initial states (lab run RN6, plan PL4). Width is the ratio of the 80% interval width to the theoretical $h$-step width; maxcellmce is the largest MCE over the processes.
  setup: studied D1; cases D2; procedure D3; lens D4, D13
  E1.T1 Case T: the true orders with the true parameters, pooled over the 11 Gaussian processes
  | row | T | D3 | initial states | D4 {nominal level: 0.8} | ratio of the 80% interval width to the theoretical width | D13 | D13 (max) |
  | r1 | 100 | T | backcasting | 0.7997 | 1.0019 | 0.002348 | 0.01451 |
  | r2 | 100 | T | optimal | 0.7972 | 0.9965 | 0.002379 | 0.01451 |
  | r3 | 300 | T | backcasting |  | 1.0007 | 0.0006806 | 0.005867 |
  | r4 | 300 | T | optimal |  | 0.9992 | 0.0008571 | 0.005867 |
  parameters: A8
  seeds: [20261105]
E2 [simulation; runs 2000] Simulation of the [[D2:known-model simulation]]: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Case S, orders selected by [[D10:AutoMSARIMA]] (run RN11, plan PL5).
  setup: studied D10; cases D2; procedure D3; lens D4
  E2.T1 Case S: orders selected by AutoMSARIMA, pooled over the 11 Gaussian processes
  | row | T | D3 | initial states | D4 {nominal level: 0.8} |
  | r1 | 100 | S | backcasting | 0.7559 |
  | r2 | 100 | S | optimal | 0.7572 |
  parameters: A8
  seeds: [20261105]
E3 [simulation; runs 2000] Simulation of the [[D2:known-model simulation]]: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Case E with the [[D1:MSARIMA analytic interval]] against the simulated interval of the same fit (`interval="simulated"`, 10,000 paths) (run RN7, plan PL6).
  setup: studied D1; cases D2; procedure D3; lens D4, D13
  E3.T1 Case E: the analytic interval against the simulated interval of the same fit, pooled over the 11 Gaussian processes
  | row | T | D3 | initial states | interval | ratio of the 80% interval width to the theoretical width | D13 |
  | r1 | 50 | E | backcasting | analytic | 0.9572 | 0.01962 |
  | r2 | 50 | E | backcasting | simulated interval | 0.957 | 0.01966 |
  | r3 | 100 | E | backcasting | analytic | 0.9807 | 0.009999 |
  | r4 | 100 | E | backcasting | simulated interval | 0.9805 | 0.01002 |
  | r5 | 300 | E | backcasting | analytic | 0.9966 | 0.002644 |
  | r6 | 300 | E | backcasting | simulated interval | 0.9964 | 0.002689 |
  parameters: A8
  seeds: [20261105]
E4 [simulation; runs 2000] Simulation of the [[D2:known-model simulation]]: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Cases T and E (run RN8, plan PL7).
  setup: studied D1; cases D2; procedure D3; lens D4
  E4.T1 Cases T and E, pooled over the 11 Gaussian processes
  | row | T | D3 | initial states | D4 {nominal level: 0.8} |
  | r1 | 100 | E | backcasting | 0.7701 |
  | r2 | 100 | T | backcasting | 0.7997 |
  parameters: A8
  seeds: [20261105]
E5 [simulation; runs 2000] Simulation of the [[D2:known-model simulation]]: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Case E with `predict()` against [[D12:reforecast]] (suffix _ref) (run RN9, plan PL8).
  setup: studied D1, D12; cases D2; procedure D3; lens D13, D14
  E5.T1 Case E with predict() and with reforecast, pooled over the 11 Gaussian processes
  | row | T | D3 | initial states | forecast | D13 | scaled pinball relative to the true model with true parameters |
  | r1 | 100 | E | backcasting | predict() | 0.009999 | 1.0432 |
  | r2 | 100 | E | backcasting | reforecast | 0.02652 | 1.9575 |
  | r3 | 100 | E | optimal | predict() | 0.008501 | 1.0578 |
  | r4 | 100 | E | optimal | reforecast | 0.005329 | 1.3961 |
  parameters: A8
  seeds: [20261105]
E6 [simulation; runs 2000] Simulation of the [[D2:known-model simulation]]: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Case W: each process fitted with one fixed wrong model (e.g. a random walk with drift fitted as ARIMA(0,1,0) without drift, an ARIMA(1,1,1) as ARIMA(0,1,1)); 12 wrong models chosen by the authors (run RN10, plan PL9).
  setup: studied D1; cases D2; procedure D3; lens D4
  E6.T1 Case W (fixed wrong models) and the true model with true parameters, pooled over the 11 Gaussian processes
  | row | T | D3 | D4 {nominal level: 0.8} | share of holdout observations at or below the 0.5 quantile |
  | r1 | 100 | W | 0.6756 | 0.4281 |
  | r2 | 100 | true model, true parameters | 0.8031 | 0.4988 |
  parameters: A8
  seeds: [20261105]
E7 [quant; design benchmark] [[D10:AutoMSARIMA]] with backcasting fitted to each of the 5,315 [[D5:benchmark]] series; `predict()` and [[D12:reforecast]] quantiles scored on the holdout, with the harness of the smooth benchmark notebook of 2026-11-05; the `predict()` coverage reproduces the saved benchmark array exactly (run RN12, plan PL10).
  setup: studied D10, D12; cases D5; lens D4, D14
  E7.T1 AutoMSARIMA with backcasting on the 5,315 benchmark series: predict() against reforecast {data: D5, initial states: backcasting}
  | row | forecast | D4 {nominal level: 0.8} | D14 (mean) |
  | r1 | predict() | 0.6812 | 0.8991 |
  | r2 | D12 | 0.6827 | 6.984e+191 |
E8 [quant; design benchmark] [[D10:AutoMSARIMA]] with backcasting and ETS(MNN) fitted to every series of the [[D6:M4 sample]]; cov80 per [[D7:level terciles]] (run RN41, plan PL12).
  setup: studied D10, D15; cases D6; procedure D7; lens D4, D13, D17
  E8.T1 AutoMSARIMA and ETS(MNN) on the M4 sample {data: D6}
  | row | method | D4 {nominal level: 0.8} | D13 | D4 {nominal level: 0.8, D7: low} | D4 {nominal level: 0.8, D7: mid} | D4 {nominal level: 0.8, D7: high} | 80% coverage in the low tercile minus that in the high tercile | share of holdout observations at or below the 0.5 quantile |
  | r1 | D10 | 0.6889 | 0.03865 | 0.8128 | 0.6995 | 0.5544 | 0.2585 |  |
  | r2 | ETS(MNN) | 0.7854 | 0.08485 | 0.764 | 0.7779 | 0.8142 | -0.05019 | 0.3723 |
E9 [quant; design benchmark] [[D10:AutoMSARIMA]] and the [[D11:log-normal ARIMA]] with the same selected orders on the [[D6:M4 sample]]; MCE and cov80 per [[D7:level terciles]] (run RN42, plan PL13).
  setup: studied D10, D11; cases D6; procedure D7; lens D4, D13, D14, D17
  E9.T1 AutoMSARIMA and the log-normal ARIMA with the same orders on the M4 sample {data: D6}
  | row | method | D13 | D14 (median) | range of the 80% coverage over the level terciles | D4 {nominal level: 0.8} | D4 {nominal level: 0.8, D7: low} | D4 {nominal level: 0.8, D7: mid} | D4 {nominal level: 0.8, D7: high} |
  | r1 | D10 | 0.03865 | 0.6752 | 0.2585 |  |  |  |  |
  | r2 | D11 | 0.01771 | 0.7679 | 0.03108 | 0.7698 | 0.7742 | 0.7832 | 0.7521 |
E10 [simulation; runs 2000] Simulation of the [[D2:known-model simulation]]: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Cases T, E, S with backcasting and E with optimal initial states, run with [[D21:smooth 1.0.8 and 1.3.0]] on the same seeds (runs RN43 for 1.0.8 and RN44 for 1.3.0, plan PL11).
  setup: studied D10; cases D2; procedure D3; lens D4, D13; context D21
  E10.T1 smooth 1.0.8 against 1.3.0 on the same seeds, pooled over the 11 Gaussian processes
  | row | T | D3 | initial states | D21 | D13 | ratio of the 80% interval width to the theoretical width | D4 {nominal level: 0.8} |
  | r1 | 100 | E | backcasting | 1.0.8 | 0.01202 |  |  |
  | r2 | 100 | E | optimal | 1.0.8 | 0.02458 | 3.3857 |  |
  | r3 | 100 | S | backcasting | 1.0.8 | 0.01724 |  |  |
  | r4 | 100 | T | backcasting | 1.0.8 |  | 1.1105 | 0.8256 |
  | r5 | 100 | E | backcasting | 1.3.0 | 0.009999 |  |  |
  | r6 | 100 | E | optimal | 1.3.0 | 0.008501 | 1.0111 |  |
  | r7 | 100 | S | backcasting | 1.3.0 | 0.01388 |  |  |
  | r8 | 100 | T | backcasting | 1.3.0 |  | 1.0019 | 0.7997 |
  parameters: A8
  seeds: [20261105]
E11 [quant; design benchmark] Planned comparison of [[D8:auto.adam error selection]] with auto.msarima on the [[D9:fresh M4 sample]] (plan PL14). Its only run, RN47, failed while loading M4 because the cached M4 files had been removed; no model was fitted and no result exists.
  setup: studied D8, D10; cases D9; procedure D7; lens D13
E12 [quant; design benchmark] R auto.msarima ([[D10:AutoMSARIMA]]) and [[D8:auto.adam error selection]] fitted to the 4,500 series of the [[D9:fresh M4 sample]] with backcasting; 99 quantiles per series scored on the M4 holdouts; MCE, cov80 by [[D7:level terciles]], scaled pinball, and the share of series choosing dlnorm (run RN48, plan PL15, which repeats PL14 after its failed run).
  setup: studied D8, D10; cases D9; procedure D7; lens D4, D13, D14, D17
  E12.T1 R auto.msarima and auto.adam error selection on the fresh M4 sample, backcasting {data: D9, initial states: backcasting}
  | row | method | D4 {nominal level: 0.8} | D13 | D14 (median) | D4 {nominal level: 0.8, D7: low} | D4 {nominal level: 0.8, D7: mid} | D4 {nominal level: 0.8, D7: high} | range of the 80% coverage over the level terciles | share of series for which dlnorm is selected |
  | r1 | D8 | 0.7461 | 0.02236 | 0.7514 | 0.7865 | 0.7425 | 0.7092 | 0.0773 | 0.6702 |
  | r2 | D10 | 0.6923 | 0.0352 | 0.6872 | 0.8193 | 0.7098 | 0.5477 | 0.2715 |  |
E13 [quant; design benchmark] Coverage by [[D7:level terciles]] on the 5,315 [[D5:benchmark]] series for [[D10:AutoMSARIMA]] with backcasting (refitted) and for the [[D16:other ARIMA implementations]] (saved forecasts of the smooth benchmark of 2026-11-05); and coverage of ETS and ARIMA on the series where the selected ETS(ANN) of [[D15:ETS forced and selected]] and the selected ARIMA(0,1,1) give the same forecast and width (exploratory run RN15).
  setup: studied D10, D16, D15; cases D5; procedure D7; lens D4, D17
  E13.T1 80% coverage by level tercile of AutoMSARIMA and the other ARIMA implementations on the benchmark {data: D5}
  | row | method | D4 {nominal level: 0.8, D7: low} | D4 {nominal level: 0.8, D7: mid} | D4 {nominal level: 0.8, D7: high} |
  | r1 | D10 | 0.7862 | 0.6677 | 0.5897 |
  | r2 | statsforecast AutoARIMA | 0.7891 | 0.6672 | 0.5893 |
  | r3 | pmdarima auto_arima | 0.7702 | 0.6354 | 0.5825 |
  | r4 | skforecast Arima | 0.7809 | 0.6578 | 0.5854 |
  E13.T2 Series of the benchmark where the selected ETS(ANN) and the selected ARIMA(0,1,1) give the same forecast and width {data: D5}
  | row | method | series | D4 {nominal level: 0.8} |
  | r1 | both | 114 |  |
  | r2 | ETS |  | 0.7336 |
  | r3 | ARIMA |  | 0.7336 |
E14 [quant; design benchmark] [[D15:ETS forced and selected]] and ARIMA(0,1,1) forced on all 5,315 [[D5:benchmark]] series and the selected ETS and [[D10:AutoMSARIMA]] models, backcasting; cov80 per method (exploratory run RN14 on the forecasts of RN13).
  setup: studied D15, D10; cases D5; lens D4
  E14.T1 80% coverage of forced and selected models on all 5,315 benchmark series, backcasting {data: D5}
  | row | method | D4 {nominal level: 0.8} |
  | r1 | ARIMA(0,1,1) | 0.6951 |
  | r2 | ETS(ANN) | 0.6681 |
  | r3 | ETS(MNN) | 0.7635 |
  | r4 | ETS, selected | 0.7492 |
  | r5 | ARIMA, selected | 0.6812 |
E15 [quant; design benchmark] [[D10:AutoMSARIMA]] and the [[D11:log-normal ARIMA]] with the same selected orders on the 5,315 [[D5:benchmark]] series, backcasting; cov80, MCE, scaled pinball and cov80 per [[D7:level terciles]] (exploratory run RN38).
  setup: studied D10, D11; cases D5; procedure D7; lens D4, D13, D14, D17
  E15.T1 AutoMSARIMA and the log-normal ARIMA with the same orders on the benchmark, backcasting {data: D5, initial states: backcasting}
  | row | method | D4 {nominal level: 0.8} | D13 | D14 (median) | D4 {nominal level: 0.8, D7: low} | D4 {nominal level: 0.8, D7: mid} | D4 {nominal level: 0.8, D7: high} | D14 (mean) |
  | r1 | D10 | 0.6812 | 0.04024 | 0.4959 | 0.7862 | 0.6677 | 0.5897 | 0.8991 |
  | r2 | D11 | 0.7563 | 0.01244 | 0.5507 | 0.7493 | 0.7514 | 0.7683 | 1.0916 |
E16 [quant; design benchmark] R auto.msarima ([[D10:AutoMSARIMA]]) and [[D8:auto.adam error selection]] on the 5,315 [[D5:benchmark]] series, backcasting; 99 quantiles per series; MCE, cov80 by [[D7:level terciles]], scaled pinball and the share choosing dlnorm (exploratory run RN45). Run in R because Python's AutoADAM removes dlnorm from the candidate distributions of a pure ARIMA whatever the user passes.
  setup: studied D8, D10; cases D5; procedure D7; lens D4, D13, D14, D17
  E16.T1 R auto.msarima and auto.adam error selection on the benchmark, backcasting {data: D5, initial states: backcasting}
  | row | method | series fitted | series that failed | D4 {nominal level: 0.8} | share of holdout observations at or below the 0.1 quantile | share of holdout observations at or below the 0.5 quantile | share of holdout observations at or below the 0.9 quantile | D13 | D14 (mean) | D14 (median) | D4 {nominal level: 0.8, D7: low} | D4 {nominal level: 0.8, D7: mid} | D4 {nominal level: 0.8, D7: high} | range of the 80% coverage over the level terciles | share of series for which dlnorm is selected |
  | r1 | D8 | 5315 | 0 | 0.7291 | 0.1421 | 0.5024 | 0.8712 | 0.01866 | 0.9225 | 0.5137 | 0.7539 | 0.7102 | 0.7233 | 0.04363 | 0.642 |
  | r2 | D10 | 5315 | 0 | 0.6812 | 0.1436 | 0.4703 | 0.8248 | 0.04024 | 0.8991 | 0.4959 | 0.7862 | 0.6674 | 0.59 | 0.1962 | 0 |
  E16.T2 The same by the error distribution selected {data: D5, initial states: backcasting}
  | row | method | selected distribution | D4 {nominal level: 0.8} | D13 |
  | r1 | D8 | dlnorm | 0.7503 | 0.02725 |
  | r2 | D8 | dnorm | 0.6912 | 0.04425 |
  | r3 | D10 | dnorm | 0.6812 | 0.04024 |
E17 [quant; design benchmark] [[D10:AutoMSARIMA]] forecasts of the 5,315 [[D5:benchmark]] series with [[D21:smooth 1.0.8 and 1.3.0]] decomposed in stages: A, 1.0.8; D, 1.3.0 with the parameters of A; B, 1.3.0 with the orders of A; C, 1.3.0. Width ratios are medians over series of the 80% interval width of the later stage over the earlier one (exploratory run RN20 on runs RN16 to RN19).
  setup: studied D10; cases D5; lens D4; context D21
  E17.T1 Stages of the change from smooth 1.0.8 to 1.3.0 on the benchmark {data: D5}
  | row | initial states | stage | median ratio of the 80% interval width of the later stage to the earlier | change of 80% coverage | share of selected models with drift |
  | r1 | backcasting | A->C | 1 | -0.01894 |  |
  | r2 | optimal | A->C | 1 |  |  |
  | r3 | backcasting | A->D | 0.9871 |  |  |
  | r4 | backcasting | C |  |  | 0.2476 |
  | r5 | backcasting | A |  |  | 0.3087 |
E18 [quant; design benchmark] 500 [[D5:benchmark]] series drawn with seed 41: R msarima with the orders Python selected, and R auto.msarima against Python [[D10:AutoMSARIMA]], backcasting; ratio of the 80% interval widths and coverage (exploratory run RN37).
  setup: studied D10; cases D5; lens D4
  E18.T1 R against Python AutoMSARIMA on 500 benchmark series drawn with seed 41, backcasting
  | row | median ratio of the R to the Python 80% interval width | share of series where R and Python select the same model | D4 {nominal level: 0.8, implementation: R} | D4 {nominal level: 0.8, implementation: Python} |
  | r1 | 1 | 1 | 0.6766 | 0.6766 |
E19 [quant; design other] Coverage the [[D2:known-model simulation]] predicts at the lengths of the 5,315 [[D5:benchmark]] series: the pooled cov80 of each case of [[D3:cases T, E, S, W]] interpolated on log T at each series length (lengths below 15 and above 300 at the nearest size) and averaged over the series (exploratory run RN40 on runs RN8, RN11, RN39).
  setup: studied D10; cases D2, D5; procedure D3; lens D4
  E19.T1 80% coverage the known-model simulation predicts at the lengths of the benchmark series {data: D5}
  | row | D3 | D4 {nominal level: 0.8} |
  | r1 | T | 0.8012 |
  | r2 | E | 0.725 |
  | r3 | S | 0.7169 |
E20 [quant; design benchmark] MCE of each calibration curve of the 99 quantiles drawn from recorded outputs: the [[D2:known-model simulation]] (runs RN8, RN11, RN39), [[D21:smooth 1.0.8 and 1.3.0]] (RN16, RN19, RN43, RN44), the [[D5:benchmark]] (RN45 and the saved ETS benchmark) and the [[D9:fresh M4 sample]] (RN48), overall and by [[D7:level terciles]] (exploratory run RN49).
  setup: studied D10, D8, D15; cases D2, D5, D9; procedure D7; lens D13
  E20.T1 MCE of each curve of figure A3
  | row | panel | line | D13 |
  | r1 | T=15 | true parameters | 0.01439 |
  | r2 | T=15 | estimated parameters | 0.06533 |
  | r3 | T=15 | selected orders | 0.05793 |
  | r4 | T=25 | true parameters | 0.006361 |
  | r5 | T=25 | estimated parameters | 0.03414 |
  | r6 | T=25 | selected orders | 0.03659 |
  | r7 | T=50 | true parameters | 0.001008 |
  | r8 | T=50 | estimated parameters | 0.01962 |
  | r9 | T=50 | selected orders | 0.02353 |
  | r10 | T=100 | true parameters | 0.002348 |
  | r11 | T=100 | estimated parameters | 0.009999 |
  | r12 | T=100 | selected orders | 0.01388 |
  | r13 | T=300 | true parameters | 0.0006806 |
  | r14 | T=300 | estimated parameters | 0.002644 |
  | r15 | T=300 | selected orders | 0.004572 |
  E20.T2 MCE of each curve of figure A4
  | row | panel | line | D13 |
  | r1 | benchmark, backcasting | 1.0.8 | 0.03258 |
  | r2 | benchmark, backcasting | 1.3.0 | 0.04024 |
  | r3 | benchmark, optimal | 1.0.8 | 0.03312 |
  | r4 | benchmark, optimal | 1.3.0 | 0.03899 |
  | r5 | simulation T=100, true parameters | 1.0.8 | 0.01229 |
  | r6 | simulation T=100, true parameters | 1.3.0 | 0.002348 |
  | r7 | simulation T=100, estimated | 1.0.8 | 0.01202 |
  | r8 | simulation T=100, estimated | 1.3.0 | 0.009999 |
  | r9 | simulation T=100, selected | 1.0.8 | 0.01724 |
  | r10 | simulation T=100, selected | 1.3.0 | 0.01388 |
  E20.T3 MCE of each curve of figure A5
  | row | panel | line | D13 |
  | r1 | all series | D10 | 0.04024 |
  | r2 | all series | D8 | 0.01866 |
  | r3 | all series | ETS (AutoADAM ZXZ, saved) | 0.04424 |
  | r4 | low level tercile | D10 | 0.0281 |
  | r5 | low level tercile | D8 | 0.0334 |
  | r6 | mid level tercile | D10 | 0.05052 |
  | r7 | mid level tercile | D8 | 0.03523 |
  | r8 | high level tercile | D10 | 0.0623 |
  | r9 | high level tercile | D8 | 0.06547 |
  | r10 | auto.adam by selected distribution | chose dnorm | 0.04425 |
  | r11 | auto.adam by selected distribution | chose dlnorm | 0.02725 |
  E20.T4 MCE of each curve of figure A6
  | row | panel | line | D13 |
  | r1 | all series | D10 | 0.0352 |
  | r2 | all series | D8 | 0.02236 |
  | r3 | low level tercile | D10 | 0.03481 |
  | r4 | low level tercile | D8 | 0.04103 |
  | r5 | mid level tercile | D10 | 0.04118 |
  | r6 | mid level tercile | D8 | 0.02004 |
  | r7 | high level tercile | D10 | 0.07568 |
  | r8 | high level tercile | D8 | 0.1052 |
  | r9 | auto.adam by selected distribution | chose dnorm | 0.02093 |
  | r10 | auto.adam by selected distribution | chose dlnorm | 0.03506 |

WARRANTS
W1 [abductive; premises C10, C11] Where ETS and ARIMA reduce to the same model, their intervals coincide and cover the same share (C17), so the interval computations agree; the additive ETS(ANN) forced on every series covers no more than ARIMA(0,1,1), while the multiplicative ETS(MNN) covers 0.07 to 0.10 more (C22). The difference between the pools is therefore the multiplicative-error models, which AutoMSARIMA does not have.

LIMITATIONS
L1 [sampling; affects C16; could_bias_magnitude; mitigation none] The 12 wrong models of case W were chosen by the authors, not drawn from the mistakes AutoMSARIMA makes, so the size of the coverage loss they produce depends on that choice.
L2 [other; affects C2, C3, C4; reduces_precision; mitigation none] The lab record flags plans PL11, PL12 and PL13 with code changed after the plan; the only commit between those plans and their runs updated the claimset skills under .claude/skills, not the analysis scripts.
L3 [measurement; affects C2; reduces_transferability; mitigation none] ETS(MNN), the comparator whose coverage stays flat over the level terciles on M4, is badly centred there (q50 of 0.372) and has an MCE of 0.0849, so its flat cov80 does not make it well calibrated.
L4 [computational; affects C4; could_bias_magnitude; mitigation none] smooth 1.0.8 ran from a separate worktree at tag v1.0.8-py with its own Python environment, so the versions also differ in the environment around the package.
L5 [computational; affects C5, C20; reduces_transferability; mitigation none] The auto.adam comparison ran in R, because Python's AutoADAM removes dlnorm from the candidate distributions of a pure ARIMA whatever the user passes, while R's auto.adam removes it only from its default list.
L6 [analytic_flexibility; affects C5; could_bias_magnitude; mitigation none] The thresholds of plan PL15 were fixed after the exploratory benchmark run of auto.adam (RN45) was seen, on M4 series that run did not use.
L7 [sample_size; affects C10; reduces_precision; mitigation none] The series where the selected ETS(ANN) and ARIMA(0,1,1) give the same forecast and width are 114 of 5,315.
L8 [model_specification; affects C21; could_bias_magnitude; mitigation none] The prediction of coverage at the benchmark lengths uses Gaussian processes and interpolates the simulated coverage over log length, with lengths outside 15 to 300 at the nearest simulated size.

OPEN QUESTIONS
O1 [from C5, C22] Why do the quantiles of auto.adam with error selection lie too high, by about 0.08 to 0.13 of coverage between the 0.2 and 0.6 quantiles, on the series of the highest level tercile, and would a model of the location of grown series correct it?
O2 [from C15, C17] Why do the parameter draws of reforecast() give explosive or several times too wide intervals for ARIMA models with a constant or AR terms, differently in R and Python?
O3 [from C3, C19, C20] Would AutoMSARIMA calibrate better with error selection than with a fixed log-normal error once the location error of grown series is addressed, given that the fixed log-normal error costs 11% of median pinball on the benchmark?
O4 [from C4, C23] Which of the changes between smooth 1.0.8 and 1.3.0 that alter model selection (fewer drift models chosen with backcasting) makes the benchmark calibration of AutoMSARIMA worse while its calibration on known models improves?

ARTEFACTS
A1 [manuscript] Written record of the investigation up to plan PL13 (questions Q1 to Q5), each number taken from a recorded run.
A2 [code] Notebook reproducing the tables of the written record from the recorded run outputs.
A3 [figure, data_plot] Calibration curves of the 99 quantiles in the known-model simulation, cases T, E and S, at T = 15, 25, 50, 100, 300: coverage of each quantile (top) and coverage minus level (bottom). shows E20.T1.r1, E20.T1.r2, E20.T1.r3, E20.T1.r4, E20.T1.r5, E20.T1.r6, E20.T1.r7, E20.T1.r8, E20.T1.r9, E20.T1.r10, E20.T1.r11, E20.T1.r12, E20.T1.r13, E20.T1.r14, E20.T1.r15
A4 [figure, data_plot] Calibration curves of the 99 quantiles of AutoMSARIMA 1.0.8 and 1.3.0 on the benchmark (backcasting, optimal) and in the simulation at T = 100. shows E20.T2.r1, E20.T2.r2, E20.T2.r3, E20.T2.r4, E20.T2.r5, E20.T2.r6, E20.T2.r7, E20.T2.r8, E20.T2.r9, E20.T2.r10
A5 [figure, data_plot] Calibration curves of the 99 quantiles on the 5,315 benchmark series: auto.msarima, auto.adam with error selection and the selected ETS, overall, by level tercile and by the distribution auto.adam chose. shows E20.T3.r1, E20.T3.r2, E20.T3.r3, E20.T3.r4, E20.T3.r5, E20.T3.r6, E20.T3.r7, E20.T3.r8, E20.T3.r9, E20.T3.r10, E20.T3.r11
A6 [figure, data_plot] Calibration curves of the 99 quantiles on the fresh M4 sample: auto.msarima and auto.adam with error selection, overall, by level tercile and by the distribution auto.adam chose. shows E20.T4.r1, E20.T4.r2, E20.T4.r3, E20.T4.r4, E20.T4.r5, E20.T4.r6, E20.T4.r7, E20.T4.r8, E20.T4.r9, E20.T4.r10
A7 [lab_record] Lab record of the study (branch claimset/msarima-interval-calibration): questions, hypotheses, plans, approvals, runs, results and notes as an append-only, hash-chained event log (lab record 0.20, claimset 0.39.0).
A8 [code] Scripts of the study: simulation (q1_*), version decomposition (q2_*), benchmark analyses (q3_*), M4 (q4_m4.py), auto.adam in R and the calibration curves (q5_*).
A9 [data] Outputs of every recorded run (per-replicate and per-series coverage, pinball and selected models; no observations of the M1, M3, Tourism or M4 series).
A10 [data] Every result recorded in the lab record (A7), one row per result: the lab result ID, the run that produced it, the quantity as the analysis script names it, its conditions as JSON and its full-precision value. The tables of the evidence objects hold these values rounded to four significant digits and at least four decimals.

REFERENCES
B1 10.1002/for.3980010202 (doi)
B2 10.1016/S0169-2070(00)00057-1 (doi)
B3 10.1016/j.ijforecast.2010.04.009 (doi)
B4 10.1016/j.ijforecast.2019.04.014 (doi)
B5 10.1080/07350015.1993.10509938 (doi)
B6 10.1201/9781003452652 (doi)

SCORES
Scores are computed by rules from the claimset file (EntryLib rubric 0.2): they check the record, not the science, and may change as the rubric and reviews develop.
Reliability of the paper: 60 (medium), the score of its lowest principal claim, C6.
C1: reliability 94 (high); novelty: new, by the authors · no close match on EntryLib
C2: reliability 83 (high); novelty: new, by the authors · no close match on EntryLib
C3: reliability 78 (medium); novelty: new, by the authors · no close match on EntryLib
C4: reliability 73 (medium); novelty: new, by the authors · no close match on EntryLib
C5: reliability 79 (medium); novelty: new, by the authors · no close match on EntryLib
C6: reliability 60 (medium); novelty: new, by the authors · no close match on EntryLib
C7: reliability 97 (high); novelty: new, by the authors
C8: reliability 97 (high); novelty: new, by the authors
C9: reliability 100 (high); novelty: new, by the authors
C10: reliability 95 (high); novelty: new, by the authors
C11: reliability 100 (high); novelty: new, by the authors
C12: reliability 100 (high)
C13: reliability 97 (high); novelty: extends earlier work, by the authors
C14: reliability 97 (high); novelty: extends earlier work, by the authors
C15: reliability 97 (high); novelty: new, by the authors
C16: reliability 85 (high); novelty: new, by the authors
C17: reliability 97 (high); novelty: new, by the authors
C18: reliability 97 (high)
C19: reliability 100 (high); novelty: new, by the authors
C20: reliability 98 (high); novelty: new, by the authors
C21: reliability 83 (high); novelty: extends earlier work, by the authors
C22: reliability 100 (high); novelty: new, by the authors
C23: reliability 100 (high); novelty: new, by the authors
Open problems in the principal claims and what they rest on:
P1 [support; up to -30; C6] Rests on argument only. To remove it: Support the claim with evidence the claimset records, or have its proof checked.
P2 [open; up to -12; C4] Limitation L4 (could bias magnitude, not mitigated): smooth 1.0.8 ran from a separate worktree at tag v1.0.8-py with its own Python environment, so the versions also differ in the environment around the package. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P3 [open; up to -12; C5] Limitation L6 (could bias magnitude, not mitigated): The thresholds of plan PL15 were fixed after the exploratory benchmark run of auto.adam (RN45) was seen, on M4 series that run did not use. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P4 [context; up to -5; C3, C4, C5, C6] A principal claim related to no earlier work. To remove it: Relate the claim to the earlier work it extends, addresses or departs from.
P5 [plan; up to -5; C2, C3, C4] The lab record flags the test: code changed after plan. To remove it: Rerun the test as planned, or explain the flag in a limitation.
P6 [plan; up to -3; C1, C2, C3, C4, C5, C7] The plan's time is anchored by a git push only. To remove it: Anchor plans with the platform, a timestamp authority (RFC 3161) or a registry.
P7 [open; up to -3; C1] Assumption C24 is not checked: The coverage of a set of series is well summarised by first averaging the holdout observations within each series and then averaging over series, all horizons together. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
P8 [open; up to -5; C2, C3, C4] Limitation L2 (reduces precision, not mitigated): The lab record flags plans PL11, PL12 and PL13 with code changed after the plan; the only commit between those plans and their runs updated the claimset skills under .claude/skills, not the analysis scripts. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P9 [open; up to -1; C2] Limitation L3 (reduces transferability, not mitigated): ETS(MNN), the comparator whose coverage stays flat over the level terciles on M4, is badly centred there (q50 of 0.372) and has an MCE of 0.0849, so its flat cov80 does not make it well calibrated. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P10 [open; up to -2; C5] Limitation L5 (reduces transferability, not mitigated): The auto.adam comparison ran in R, because Python's AutoADAM removes dlnorm from the candidate distributions of a pure ARIMA whatever the user passes, while R's auto.adam removes it only from its default list. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P11 [open; up to -5; C10] Limitation L7 (reduces precision, not mitigated): The series where the selected ETS(ANN) and ARIMA(0,1,1) give the same forecast and width are 114 of 5,315. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P12 [open; up to -3; C2, C8] Assumption C25 is not checked: The ratio of the mean of the last third of a series to its overall mean indexes how much the scale of its errors at the forecast origin exceeds the average scale over the sample. To remove it: Test the assumption, or analyse how sensitive the claim is to it.

HOW TO CITE
This claimset: Svetunkov, I. (guarantor) (2026). Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4. EntryLib. entrylib:2026.000001@v2. https://entrylib.org/p/2026.000001@v2
One claim of it: by its identifier, as entrylib:2026.000001@v2#C1 for claim C1. Each principal claim has its own page, as https://entrylib.org/p/2026.000001/C1.