CLAIM C4 of entrylib:2026.000001 Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4
licence CC-BY-4.0; contributions C4

DEFINITIONS
D1 [method] MSARIMA analytic interval: The quantiles of `predict(interval="prediction")` of a pure ARIMA in smooth 1.3.0, which dispatches to the approximate interval. The ARIMA is the state space model
$$\begin{aligned} y_t &= w^\top v_{t-l} + \epsilon_t, \\ v_t &= F v_{t-l} + g \epsilon_t, \\ \epsilon_t &\sim N(0, \sigma^2). \end{aligned}$$
The quantile at level $\tau$ and horizon $h$ is
$$q_{\tau,h} = \hat y_{T+h|T} + \Phi^{-1}(\tau) \sqrt{V_h}, \qquad V_h = \hat\sigma^2 \frac{T}{T-k} c_h,$$
with
$$\hat\sigma^2 = \frac{1}{T} \sum_{t=1}^{T} e_t^2, \qquad c_h = 1 + \sum_{j=1}^{h-1} \left(w^\top F^{j-1} g\right)^2,$$
where $k$ is the number of estimated parameters and $c_h$ the $h$-th diagonal element of the analytic covariance of the state space model, written here for a model with lag 1. [source B6]
D2 [data] known-model simulation: Gaussian ARIMA series with $\sigma = 10$ generated from 11 data generating processes (white noise with a constant, random walk with and without drift, AR(1), MA(1), ARIMA(0,1,1), ARIMA(1,1,0), ARIMA(1,1,1), ARIMA(0,2,2), seasonal SARIMA(0,1,1)(0,1,1)$_{12}$ and SARIMA(1,0,0)(1,0,0)$_{12}$) plus an IMA(1,1) with Student-t errors on 4 degrees of freedom; in-sample lengths $T$ of 15, 25, 50, 100 and 300, holdout of 18; 2,000 replicates per process and length with seed 20261105 unless stated.
D3 [design] cases T, E, S, W: What the fitted AutoMSARIMA or MSARIMA knows about the data generating process of the [[D2:known-model simulation]]: T, true orders and true AR, MA and constant parameters with initial states and scale estimated; E, true orders with estimated parameters; S, orders selected by AutoMSARIMA; W, a fixed wrong model for each process. Each with backcasting or optimal initial states.
D4 [variable] coverage: For the quantile $q_{\tau,j}$ at level $\tau$ and horizon $j$, the share of holdout observations at or below it,
$$c_\tau = \frac{1}{h} \sum_{j=1}^{h} 1\left(y_{T+j} \le q_{\tau,j}\right),$$
averaged first within a series and then over series. cov80 is $c_{0.9} - c_{0.1}$, the coverage of the central 80% interval; q50 is $c_{0.5}$.
D5 [data] benchmark: The 5,315 series of M1 (1,001), M3 (3,003) and Tourism (1,311) from fcompdata 0.1.4, with their own holdouts and seasonal periods; lags $(1, m)$ for seasonal series. [members D18, D19, D20]
D6 [data] M4 sample: 1,500 series each of the yearly, quarterly and monthly M4 series, drawn with seed 20261105 without replacement, with the M4 holdouts. [source B4]
D7 [design] level terciles: Series split into terciles of the [[D17:level ratio]] across the series of a data set; cov80 reported for each tercile, and the spread as the largest minus the smallest of the three.
D8 [method] auto.adam error selection: R `auto.adam(y, model="NNN", lags, distribution=c("dnorm","dlnorm"), initial="backcasting")` of smooth 4.6: the ARIMA orders and constant are selected by AICc for each distribution with the maxima of auto.msarima (AR (3,3), I (2,1), MA (3,3)), then the distribution with the lower AICc is kept; with dnorm the model is the [[D10:AutoMSARIMA]] model, with dlnorm it is the [[D11:log-normal ARIMA]]. Quantiles from `forecast(interval="prediction")`. [source B6]
D9 [data] fresh M4 sample: 1,500 series each of the yearly, quarterly and monthly M4 series, drawn with seed 20261107 from the series not in the [[D6:M4 sample]], so disjoint from it. [source B4]
D10 [method] AutoMSARIMA: `AutoMSARIMA(lags, initial)` in Python and `auto.msarima()` in R of smooth 1.3.0 / 4.6: ARIMA orders up to AR (3,3), I (2,1), MA (3,3) and a constant selected by AICc, normal errors, the [[D1:MSARIMA analytic interval]] for the quantiles. Backcasting initial states unless stated. [source B6]
D11 [method] log-normal ARIMA: ADAM with `model="NNN"`, an ARIMA and `distribution="dlnorm"`: the ARIMA applies to the logarithms,
$$\begin{aligned} \log y_t &= w^\top v_{t-l} + \log(1+\epsilon_t), \\ v_t &= F v_{t-l} + g \log(1+\epsilon_t), \\ 1+\epsilon_t &\sim \mathrm{logN}(-\sigma^2/2, \sigma^2), \end{aligned}$$
so the scale of the error is proportional to the level. In the comparisons with fixed orders, the orders and constant are those [[D10:AutoMSARIMA]] selects on the same series. [source B6]
D13 [variable] MCE: Mean calibration error over the 99 quantile levels $\tau = 0.01, \dots, 0.99$:
$$\mathrm{MCE} = \frac{1}{99} \sum_{\tau} \left|\bar c_\tau - \tau\right|,$$
with $\bar c_\tau$ the [[D4:coverage]] of $q_\tau$.
D15 [method] ETS forced and selected: ETS of smooth with backcasting: AutoADAM-style selection among ETS models (`ES(model="ZXZ")`, or ZXN for non-seasonal series), or ETS(ANN) or ETS(MNN) fitted to every series. [source B6]
D17 [variable] level ratio: Mean of the last third of the in-sample data of a series divided by the mean of all its in-sample data.
D18 [data] M1: The 1,001 series of the M1 competition with their holdouts. [source B1]
D19 [data] M3: The 3,003 series of the M3 competition with their holdouts. [source B2]
D20 [data] Tourism: The 1,311 series of the tourism forecasting competition with their holdouts. [source B3]
D21 [setting] smooth 1.0.8 and 1.3.0: Python smooth at tag v1.0.8-py (a separate worktree with its own environment) and at 1.3.0 (commit 634657d8 of openforecast-org/smooth).

CLAIMS
C4 [descriptive; principal; support: simulation E10 (E10.T1.r1, E10.T1.r5, E10.T1.r3, E10.T1.r7, E10.T1.r2, E10.T1.r6) artefacts A4, A8; uses C9; limitations L2, L4; strength high (author); verifiability open; planning confirmatory; outcome supported]
  On known models, the [[D10:AutoMSARIMA]] interval of smooth 1.3.0 is calibrated no worse than that of 1.0.8 ([[D21:smooth 1.0.8 and 1.3.0]]): at $T = 100$ in the [[D2:known-model simulation]] its [[D13:MCE]] is 0.0100 against 0.0120 with estimated parameters, 0.0139 against 0.0172 with selected orders and 0.0085 against 0.0246 with optimal initial states.
  scope: data_type: simulated Gaussian ARIMA and SARIMA series | conditions: holdout of 18; T = 100; cases E and S
  test: deterministic; sample 2000 replicates per cell, 11 DGPs, T=50,100,300, seed 20261105 in both versions; run 1 smooth 1.0.8 (v1.0.8-py), run 2 smooth 1.3.0; supported if eb130 <= eb108+0.001 and sb130 <= sb108+0.001 and eo130 <= eo108+0.001; not supported if eb130 > eb108+0.005 or sb130 > sb108+0.005 or eo130 > eo108+0.005; verdict computed; flags code_changed_after_plan
  plan: lab plan PL11; planned 2026-11-05T18:40:15Z; anchor git_remote
C9 [descriptive; supporting; support: empirical_quant E17 (E17.T1.r1, E17.T1.r2); strength moderate (author); verifiability open; planning exploratory]
  The width of the [[D10:AutoMSARIMA]] interval on the [[D5:benchmark]] did not change between [[D21:smooth 1.0.8 and 1.3.0]]: the median ratio of the 80% widths of 1.3.0 over 1.0.8 is 1.000 with backcasting and with optimal initial states.
  scope: data_type: M1, M3 and Tourism series

EVIDENCE
E10 [simulation; runs 2000] Simulation of the [[D2:known-model simulation]]: in each replicate the model is fitted to the first T observations and its 99 quantiles (0.01 to 0.99) are scored on the 18 holdout observations; results are pooled over the 11 Gaussian processes (mean over the processes of the mean over replicates). Cases T, E, S with backcasting and E with optimal initial states, run with [[D21:smooth 1.0.8 and 1.3.0]] on the same seeds (runs RN43 for 1.0.8 and RN44 for 1.3.0, plan PL11).
  setup: studied D10; cases D2; procedure D3; lens D4, D13; context D21
  E10.T1 smooth 1.0.8 against 1.3.0 on the same seeds, pooled over the 11 Gaussian processes
  | row | T | D3 | initial states | D21 | D13 | ratio of the 80% interval width to the theoretical width | D4 {nominal level: 0.8} |
  | r1 | 100 | E | backcasting | 1.0.8 | 0.012016197326803405 |  |  |
  | r2 | 100 | E | optimal | 1.0.8 | 0.02458198143046628 | 3.3857 |  |
  | r3 | 100 | S | backcasting | 1.0.8 | 0.017238598102234465 |  |  |
  | r4 | 100 | T | backcasting | 1.0.8 |  | 1.1105 | 0.8256 |
  | r5 | 100 | E | backcasting | 1.3.0 | 0.00999872461993675 |  |  |
  | r6 | 100 | E | optimal | 1.3.0 | 0.00850140291806958 | 1.0111 |  |
  | r7 | 100 | S | backcasting | 1.3.0 | 0.01387776757473727 |  |  |
  | r8 | 100 | T | backcasting | 1.3.0 |  | 1.0019 | 0.7997 |
  parameters: A8
  seeds: [20261105]
E17 [quant; design benchmark] [[D10:AutoMSARIMA]] forecasts of the 5,315 [[D5:benchmark]] series with [[D21:smooth 1.0.8 and 1.3.0]] decomposed in stages: A, 1.0.8; D, 1.3.0 with the parameters of A; B, 1.3.0 with the orders of A; C, 1.3.0. Width ratios are medians over series of the 80% interval width of the later stage over the earlier one (exploratory run RN20 on runs RN16 to RN19).
  setup: studied D10; cases D5; lens D4; context D21
  E17.T1 Stages of the change from smooth 1.0.8 to 1.3.0 on the benchmark {data: D5}
  | row | initial states | stage | median ratio of the 80% interval width of the later stage to the earlier | change of 80% coverage | share of selected models with drift |
  | r1 | backcasting | A->C | 1 | -0.01894 |  |
  | r2 | optimal | A->C | 1 |  |  |
  | r3 | backcasting | A->D | 0.9871 |  |  |
  | r4 | backcasting | C |  |  | 0.2476 |
  | r5 | backcasting | A |  |  | 0.3087 |
E20 [quant; design benchmark] MCE of each calibration curve of the 99 quantiles drawn from recorded outputs: the [[D2:known-model simulation]] (runs RN8, RN11, RN39), [[D21:smooth 1.0.8 and 1.3.0]] (RN16, RN19, RN43, RN44), the [[D5:benchmark]] (RN45 and the saved ETS benchmark) and the [[D9:fresh M4 sample]] (RN48), overall and by [[D7:level terciles]] (exploratory run RN49).
  setup: studied D10, D8, D15; cases D2, D5, D9; procedure D7; lens D13
  E20.T1 MCE of each curve of figure A3
  | row | panel | line | D13 |
  | r1 | T=15 | true parameters | 0.01439 |
  | r2 | T=15 | estimated parameters | 0.06533 |
  | r3 | T=15 | selected orders | 0.05793 |
  | r4 | T=25 | true parameters | 0.006361 |
  | r5 | T=25 | estimated parameters | 0.03414 |
  | r6 | T=25 | selected orders | 0.03659 |
  | r7 | T=50 | true parameters | 0.001008 |
  | r8 | T=50 | estimated parameters | 0.01962 |
  | r9 | T=50 | selected orders | 0.02353 |
  | r10 | T=100 | true parameters | 0.002348 |
  | r11 | T=100 | estimated parameters | 0.009999 |
  | r12 | T=100 | selected orders | 0.01388 |
  | r13 | T=300 | true parameters | 0.0006806 |
  | r14 | T=300 | estimated parameters | 0.002644 |
  | r15 | T=300 | selected orders | 0.004572 |
  E20.T2 MCE of each curve of figure A4
  | row | panel | line | D13 |
  | r1 | benchmark, backcasting | 1.0.8 | 0.03258 |
  | r2 | benchmark, backcasting | 1.3.0 | 0.04024 |
  | r3 | benchmark, optimal | 1.0.8 | 0.03312 |
  | r4 | benchmark, optimal | 1.3.0 | 0.03899 |
  | r5 | simulation T=100, true parameters | 1.0.8 | 0.01229 |
  | r6 | simulation T=100, true parameters | 1.3.0 | 0.002348 |
  | r7 | simulation T=100, estimated | 1.0.8 | 0.01202 |
  | r8 | simulation T=100, estimated | 1.3.0 | 0.009999 |
  | r9 | simulation T=100, selected | 1.0.8 | 0.01724 |
  | r10 | simulation T=100, selected | 1.3.0 | 0.01388 |
  E20.T3 MCE of each curve of figure A5
  | row | panel | line | D13 |
  | r1 | all series | D10 | 0.04024 |
  | r2 | all series | D8 | 0.01866 |
  | r3 | all series | ETS (AutoADAM ZXZ, saved) | 0.04424 |
  | r4 | low level tercile | D10 | 0.0281 |
  | r5 | low level tercile | D8 | 0.0334 |
  | r6 | mid level tercile | D10 | 0.05052 |
  | r7 | mid level tercile | D8 | 0.03523 |
  | r8 | high level tercile | D10 | 0.0623 |
  | r9 | high level tercile | D8 | 0.06547 |
  | r10 | auto.adam by selected distribution | chose dnorm | 0.04425 |
  | r11 | auto.adam by selected distribution | chose dlnorm | 0.02725 |
  E20.T4 MCE of each curve of figure A6
  | row | panel | line | D13 |
  | r1 | all series | D10 | 0.0352 |
  | r2 | all series | D8 | 0.02236 |
  | r3 | low level tercile | D10 | 0.03481 |
  | r4 | low level tercile | D8 | 0.04103 |
  | r5 | mid level tercile | D10 | 0.04118 |
  | r6 | mid level tercile | D8 | 0.02004 |
  | r7 | high level tercile | D10 | 0.07568 |
  | r8 | high level tercile | D8 | 0.1052 |
  | r9 | auto.adam by selected distribution | chose dnorm | 0.02093 |
  | r10 | auto.adam by selected distribution | chose dlnorm | 0.03506 |

LIMITATIONS
L2 [other; affects C2, C3, C4; reduces_precision; mitigation none] The lab record flags plans PL11, PL12 and PL13 with code changed after the plan; the only commit between those plans and their runs updated the claimset skills under .claude/skills, not the analysis scripts.
L4 [computational; affects C4; could_bias_magnitude; mitigation none] smooth 1.0.8 ran from a separate worktree at tag v1.0.8-py with its own Python environment, so the versions also differ in the environment around the package.

ARTEFACTS
A3 [figure, data_plot] Calibration curves of the 99 quantiles in the known-model simulation, cases T, E and S, at T = 15, 25, 50, 100, 300: coverage of each quantile (top) and coverage minus level (bottom). shows E20.T1.r1, E20.T1.r2, E20.T1.r3, E20.T1.r4, E20.T1.r5, E20.T1.r6, E20.T1.r7, E20.T1.r8, E20.T1.r9, E20.T1.r10, E20.T1.r11, E20.T1.r12, E20.T1.r13, E20.T1.r14, E20.T1.r15
A4 [figure, data_plot] Calibration curves of the 99 quantiles of AutoMSARIMA 1.0.8 and 1.3.0 on the benchmark (backcasting, optimal) and in the simulation at T = 100. shows E20.T2.r1, E20.T2.r2, E20.T2.r3, E20.T2.r4, E20.T2.r5, E20.T2.r6, E20.T2.r7, E20.T2.r8, E20.T2.r9, E20.T2.r10
A5 [figure, data_plot] Calibration curves of the 99 quantiles on the 5,315 benchmark series: auto.msarima, auto.adam with error selection and the selected ETS, overall, by level tercile and by the distribution auto.adam chose. shows E20.T3.r1, E20.T3.r2, E20.T3.r3, E20.T3.r4, E20.T3.r5, E20.T3.r6, E20.T3.r7, E20.T3.r8, E20.T3.r9, E20.T3.r10, E20.T3.r11
A6 [figure, data_plot] Calibration curves of the 99 quantiles on the fresh M4 sample: auto.msarima and auto.adam with error selection, overall, by level tercile and by the distribution auto.adam chose. shows E20.T4.r1, E20.T4.r2, E20.T4.r3, E20.T4.r4, E20.T4.r5, E20.T4.r6, E20.T4.r7, E20.T4.r8, E20.T4.r9, E20.T4.r10
A7 [lab_record] Lab record of the study (branch claimset/msarima-interval-calibration): questions, hypotheses, plans, approvals, runs, results and notes as an append-only, hash-chained event log (lab record 0.20, claimset 0.39.0).
A8 [code] Scripts of the study: simulation (q1_*), version decomposition (q2_*), benchmark analyses (q3_*), M4 (q4_m4.py), auto.adam in R and the calibration curves (q5_*).
A10 [data] Every result recorded in the lab record (A7), one row per result: the lab result ID, the run that produced it, the quantity as the analysis script names it, its conditions as JSON and its full-precision value. The tables of the evidence objects hold these values rounded to four significant digits and at least four decimals.

REFERENCES
B1 10.1002/for.3980010202 (doi)
B2 10.1016/S0169-2070(00)00057-1 (doi)
B3 10.1016/j.ijforecast.2010.04.009 (doi)
B4 10.1016/j.ijforecast.2019.04.014 (doi)
B6 10.1201/9781003452652 (doi)

HOW TO CITE
This claim: Svetunkov, I. (guarantor) (2026). Claim C4, in: Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4. EntryLib. entrylib:2026.000001@v3#C4. https://entrylib.org/p/2026.000001/C4
The claimset it is from: Svetunkov, I. (guarantor) (2026). Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4. EntryLib. entrylib:2026.000001@v3. https://entrylib.org/p/2026.000001@v3