CLAIM C3 of entrylib:2026.000001 Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4
licence CC-BY-4.0; contributions C3

DEFINITIONS
D1 [method] MSARIMA analytic interval: The quantiles of `predict(interval="prediction")` of a pure ARIMA in smooth 1.3.0, which dispatches to the approximate interval. The ARIMA is the state space model
$$\begin{aligned} y_t &= w^\top v_{t-l} + \epsilon_t, \\ v_t &= F v_{t-l} + g \epsilon_t, \\ \epsilon_t &\sim N(0, \sigma^2). \end{aligned}$$
The quantile at level $\tau$ and horizon $h$ is
$$q_{\tau,h} = \hat y_{T+h|T} + \Phi^{-1}(\tau) \sqrt{V_h}, \qquad V_h = \hat\sigma^2 \frac{T}{T-k} c_h,$$
with
$$\hat\sigma^2 = \frac{1}{T} \sum_{t=1}^{T} e_t^2, \qquad c_h = 1 + \sum_{j=1}^{h-1} \left(w^\top F^{j-1} g\right)^2,$$
where $k$ is the number of estimated parameters and $c_h$ the $h$-th diagonal element of the analytic covariance of the state space model, written here for a model with lag 1. [source B6]
D4 [variable] coverage: For the quantile $q_{\tau,j}$ at level $\tau$ and horizon $j$, the share of holdout observations at or below it,
$$c_\tau = \frac{1}{h} \sum_{j=1}^{h} 1\left(y_{T+j} \le q_{\tau,j}\right),$$
averaged first within a series and then over series. cov80 is $c_{0.9} - c_{0.1}$, the coverage of the central 80% interval; q50 is $c_{0.5}$.
D5 [data] benchmark: The 5,315 series of M1 (1,001), M3 (3,003) and Tourism (1,311) from fcompdata 0.1.4, with their own holdouts and seasonal periods; lags $(1, m)$ for seasonal series. [members D18, D19, D20]
D6 [data] M4 sample: 1,500 series each of the yearly, quarterly and monthly M4 series, drawn with seed 20261105 without replacement, with the M4 holdouts. [source B4]
D7 [design] level terciles: Series split into terciles of the [[D17:level ratio]] across the series of a data set; cov80 reported for each tercile, and the spread as the largest minus the smallest of the three.
D10 [method] AutoMSARIMA: `AutoMSARIMA(lags, initial)` in Python and `auto.msarima()` in R of smooth 1.3.0 / 4.6: ARIMA orders up to AR (3,3), I (2,1), MA (3,3) and a constant selected by AICc, normal errors, the [[D1:MSARIMA analytic interval]] for the quantiles. Backcasting initial states unless stated. [source B6]
D11 [method] log-normal ARIMA: ADAM with `model="NNN"`, an ARIMA and `distribution="dlnorm"`: the ARIMA applies to the logarithms,
$$\begin{aligned} \log y_t &= w^\top v_{t-l} + \log(1+\epsilon_t), \\ v_t &= F v_{t-l} + g \log(1+\epsilon_t), \\ 1+\epsilon_t &\sim \mathrm{logN}(-\sigma^2/2, \sigma^2), \end{aligned}$$
so the scale of the error is proportional to the level. In the comparisons with fixed orders, the orders and constant are those [[D10:AutoMSARIMA]] selects on the same series. [source B6]
D13 [variable] MCE: Mean calibration error over the 99 quantile levels $\tau = 0.01, \dots, 0.99$:
$$\mathrm{MCE} = \frac{1}{99} \sum_{\tau} \left|\bar c_\tau - \tau\right|,$$
with $\bar c_\tau$ the [[D4:coverage]] of $q_\tau$.
D14 [variable] scaled pinball: Pinball loss averaged over the 99 quantile levels and the horizons, divided by the mean absolute first difference of the in-sample data for real series; in the simulation, by $\sigma$, and relpin is its ratio to the pinball of the true predictive distribution.
D15 [method] ETS forced and selected: ETS of smooth with backcasting: AutoADAM-style selection among ETS models (`ES(model="ZXZ")`, or ZXN for non-seasonal series), or ETS(ANN) or ETS(MNN) fitted to every series. [source B6]
D16 [method] other ARIMA implementations: statsforecast AutoARIMA, pmdarima auto_arima and skforecast Arima, as run in the notebook of the smooth benchmark of 2026-11-05, whose saved forecasts were read.
D17 [variable] level ratio: Mean of the last third of the in-sample data of a series divided by the mean of all its in-sample data.
D18 [data] M1: The 1,001 series of the M1 competition with their holdouts. [source B1]
D19 [data] M3: The 3,003 series of the M3 competition with their holdouts. [source B2]
D20 [data] Tourism: The 1,311 series of the tourism forecasting competition with their holdouts. [source B3]

CLAIMS
C3 [descriptive; principal; support: empirical_quant E9 (E9.T1.r2, E9.T1.r1); uses C2; limitations L2; strength moderate (author); verifiability open; planning confirmatory; outcome supported]
  On the [[D6:M4 sample]], the [[D11:log-normal ARIMA]] with the orders [[D10:AutoMSARIMA]] selects removes the fall of 80% [[D4:coverage]] over the [[D7:level terciles]] (0.774, 0.783 and 0.752, a spread of 0.031) and lowers [[D13:MCE]] from 0.0386 to 0.0177.
  scope: data_type: M4 yearly, quarterly and monthly series | conditions: backcasting; orders fixed at those AutoMSARIMA selects
  test: deterministic; sample 4500 M4 series as in the plan of H8, same seed; supported if sd <= 0.05 and md <= 0.5*ma; not supported if sd > 0.10 or md >= ma; verdict computed; flags code_changed_after_plan
  plan: lab plan PL13; planned 2026-11-05T18:40:25Z; anchor git_remote
C2 [descriptive; principal; support: empirical_quant E8 (E8.T1.r1, E8.T1.r2); assumes C25; uses C8; limitations L2, L3; strength moderate (author); verifiability open; planning confirmatory; outcome supported]
  On the [[D6:M4 sample]], the 80% [[D4:coverage]] of [[D10:AutoMSARIMA]] falls with the [[D17:level ratio]]: 0.813, 0.700 and 0.554 from the lowest to the highest of the [[D7:level terciles]], while ETS(MNN) of [[D15:ETS forced and selected]], whose error scale follows the level, covers 0.764, 0.778 and 0.814.
  scope: data_type: M4 yearly, quarterly and monthly series | conditions: backcasting; smooth 1.3.0 (Python)
  addresses B5: Chatfield discusses why intervals are too narrow in general; this locates the shortfall of one implementation on series whose level has grown.
  test: deterministic; sample 4500 M4 series (1500 yearly, quarterly, monthly), seed 20261105, terciles of the recent level over the mean level from the first method; supported if fa >= 0.10 and fm <= 0.05; not supported if fa < 0.05; verdict computed; flags code_changed_after_plan
  plan: lab plan PL12; planned 2026-11-05T18:40:25Z; anchor git_remote
C8 [descriptive; supporting; support: empirical_quant E13 (E13.T1.r1, E13.T1.r2, E13.T1.r3, E13.T1.r4); assumes C25; strength moderate (author); verifiability open; planning exploratory]
  On the 5,315 [[D5:benchmark]] series, the fall of 80% [[D4:coverage]] over the [[D7:level terciles]] is shared by [[D10:AutoMSARIMA]] (0.786, 0.668, 0.590) and the [[D16:other ARIMA implementations]]: statsforecast AutoARIMA (0.789, 0.667, 0.589), pmdarima (0.770, 0.635, 0.582) and skforecast (0.781, 0.658, 0.585).
  scope: data_type: M1, M3 and Tourism series | conditions: backcasting for smooth
C25 [descriptive; supporting; assumption (data, unchecked); verifiability not_applicable]
  The ratio of the mean of the last third of a series to its overall mean indexes how much the scale of its errors at the forecast origin exceeds the average scale over the sample.

EVIDENCE
E8 [quant; design benchmark] [[D10:AutoMSARIMA]] with backcasting and ETS(MNN) fitted to every series of the [[D6:M4 sample]]; cov80 per [[D7:level terciles]] (run RN41, plan PL12).
  setup: studied D10, D15; cases D6; procedure D7; lens D4, D13, D17
  E8.T1 AutoMSARIMA and ETS(MNN) on the M4 sample {data: D6}
  | row | method | D4 {nominal level: 0.8} | D13 | D4 {nominal level: 0.8, D7: low} | D4 {nominal level: 0.8, D7: mid} | D4 {nominal level: 0.8, D7: high} | 80% coverage in the low tercile minus that in the high tercile | share of holdout observations at or below the 0.5 quantile |
  | r1 | D10 | 0.6889 | 0.03865 | 0.8128 | 0.6995 | 0.5544 | 0.25849074074074074 |  |
  | r2 | ETS(MNN) | 0.7854 | 0.08485 | 0.764 | 0.7779 | 0.8142 | -0.05018518518518522 | 0.3723 |
E9 [quant; design benchmark] [[D10:AutoMSARIMA]] and the [[D11:log-normal ARIMA]] with the same selected orders on the [[D6:M4 sample]]; MCE and cov80 per [[D7:level terciles]] (run RN42, plan PL13).
  setup: studied D10, D11; cases D6; procedure D7; lens D4, D13, D14, D17
  E9.T1 AutoMSARIMA and the log-normal ARIMA with the same orders on the M4 sample {data: D6}
  | row | method | D13 | D14 (median) | range of the 80% coverage over the level terciles | D4 {nominal level: 0.8} | D4 {nominal level: 0.8, D7: low} | D4 {nominal level: 0.8, D7: mid} | D4 {nominal level: 0.8, D7: high} |
  | r1 | D10 | 0.03864774286070695 | 0.6752 | 0.2585 |  |  |  |  |
  | r2 | D11 | 0.017709097144281392 | 0.7679 | 0.03108333333333335 | 0.7698 | 0.7742 | 0.7832 | 0.7521 |
E13 [quant; design benchmark] Coverage by [[D7:level terciles]] on the 5,315 [[D5:benchmark]] series for [[D10:AutoMSARIMA]] with backcasting (refitted) and for the [[D16:other ARIMA implementations]] (saved forecasts of the smooth benchmark of 2026-11-05); and coverage of ETS and ARIMA on the series where the selected ETS(ANN) of [[D15:ETS forced and selected]] and the selected ARIMA(0,1,1) give the same forecast and width (exploratory run RN15).
  setup: studied D10, D16, D15; cases D5; procedure D7; lens D4, D17
  E13.T1 80% coverage by level tercile of AutoMSARIMA and the other ARIMA implementations on the benchmark {data: D5}
  | row | method | D4 {nominal level: 0.8, D7: low} | D4 {nominal level: 0.8, D7: mid} | D4 {nominal level: 0.8, D7: high} |
  | r1 | D10 | 0.7862 | 0.6677 | 0.5897 |
  | r2 | statsforecast AutoARIMA | 0.7891 | 0.6672 | 0.5893 |
  | r3 | pmdarima auto_arima | 0.7702 | 0.6354 | 0.5825 |
  | r4 | skforecast Arima | 0.7809 | 0.6578 | 0.5854 |
  E13.T2 Series of the benchmark where the selected ETS(ANN) and the selected ARIMA(0,1,1) give the same forecast and width {data: D5}
  | row | method | series | D4 {nominal level: 0.8} |
  | r1 | both | 114 |  |
  | r2 | ETS |  | 0.7336 |
  | r3 | ARIMA |  | 0.7336 |

LIMITATIONS
L2 [other; affects C2, C3, C4; reduces_precision; mitigation none] The lab record flags plans PL11, PL12 and PL13 with code changed after the plan; the only commit between those plans and their runs updated the claimset skills under .claude/skills, not the analysis scripts.
L3 [measurement; affects C2; reduces_transferability; mitigation none] ETS(MNN), the comparator whose coverage stays flat over the level terciles on M4, is badly centred there (q50 of 0.372) and has an MCE of 0.0849, so its flat cov80 does not make it well calibrated.

ARTEFACTS
A7 [lab_record] Lab record of the study (branch claimset/msarima-interval-calibration): questions, hypotheses, plans, approvals, runs, results and notes as an append-only, hash-chained event log (lab record 0.20, claimset 0.39.0).
A10 [data] Every result recorded in the lab record (A7), one row per result: the lab result ID, the run that produced it, the quantity as the analysis script names it, its conditions as JSON and its full-precision value. The tables of the evidence objects hold these values rounded to four significant digits and at least four decimals.

REFERENCES
B1 10.1002/for.3980010202 (doi)
B2 10.1016/S0169-2070(00)00057-1 (doi)
B3 10.1016/j.ijforecast.2010.04.009 (doi)
B4 10.1016/j.ijforecast.2019.04.014 (doi)
B6 10.1201/9781003452652 (doi)

HOW TO CITE
This claim: Svetunkov, I. (guarantor) (2026). Claim C3, in: Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4. EntryLib. entrylib:2026.000001@v3#C3. https://entrylib.org/p/2026.000001/C3
The claimset it is from: Svetunkov, I. (guarantor) (2026). Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4. EntryLib. entrylib:2026.000001@v3. https://entrylib.org/p/2026.000001@v3