Menu

You are reading a research paper published on EntryLib as separate claims, each with its evidence. How to read this page

corrected · Version 2 · 13 November 2026 · Management Science and Operations Research

Corrected · 13 November 2026 at 12:34 UTC

The paper had no summary and did not state why the ARMA part of smooth's TBATS differs from De Livera's in effect: a summary in plain language is added; a formal claim (C21, with its condition C22 and proof E6) shows that ARMA errors pass the ARMA's persistence into the level, while ADAM's additive ARMA component leaves the level a fixed share of each shock; the highlights are amended; and the equation of the harmonic states in D1 is set on a line of its own.

By Ivan Svetunkov (ORCID 0000-0001-7826-0281)

TBATS in smooth: its formulation, and its accuracy against forecast::tbats and AutoTBATS on hourly data

Responsible researcher: Ivan Svetunkov (0000-0001-7826-0281)

Written with: AI agentClaude Code · cloud session (lab actor AC1), AI agentClaude Code · session on the guarantor's computer (lab actor AC3)

CC-BY-4.0 · Keywords: TBATS, exponential smoothing, multiple seasonality, state space models, forecasting software, forecast evaluation

Made with claimset-lab. Lab record

Copies a message for ChatGPT, Claude or Gemini: paste it there.

The message

Read https://entrylib.org/p/[email protected], a research paper written as claims with their evidence. Give me a brief summary of its main findings and how well each is supported. If you cannot open the link, tell me instead of guessing. Then wait for my questions.

Summary

TBATS forecasts series with several seasonal cycles, such as hourly electricity use with daily and weekly patterns. The smooth package for R and Python builds TBATS inside its general state space framework. Simulated checks show that, without ARMA terms, it is the same model as the original TBATS of De Livera, Hyndman and Snyder to rounding error; its ARMA terms differ, adding a separate component instead of modelling the errors. On the 414 hourly series of the M4 competition, forecasting 48 hours ahead, smooth's TBATS ranked as more accurate than the TBATS of R's forecast package and of Python's statsforecast, and took about 58% of the forecast package's time. On one week of 370 hourly electricity series, it was more accurate than repeating the last week, but not shown to differ from the other two. Assuming normally distributed errors made it faster, with intervals that covered fewer values than intended.

Highlights

  • DefinitionDsmooth TBATS: TBATS of the smooth package (R tbats(), Python TBATS): a linear single-source-of-error state space model of the Box-Cox transform yt(λ)y_t^{(\lambda)} of the series, built on ADAM's state space core, yt(λ)=w⊤vt−𝒍+ϵt,vt=Fvt−𝒍+gϵt,\begin{aligned} y_t^{(\lambda)} &= w^\top v_{t-\boldsymbol{l}} + \epsilon_t, \\ v_t &= F v_{t-\boldsymbol{l}} + g \epsilon_t, \end{aligned} where vt−𝒍v_{t-\boldsymbol{l}} takes each state at its own lag. The level ltl_t and the trend btb_t are those of ETS(A,N,N), ETS(A,A,N) or ETS(A,Ad,N), lt=lt−1+ϕbt−1+αϵt,bt=ϕbt−1+βϵt.\begin{aligned} l_t &= l_{t-1} + \phi b_{t-1} + \alpha \epsilon_t, \\ b_t &= \phi b_{t-1} + \beta \epsilon_t. \end{aligned} Each harmonic j=1,…,kij = 1, \dots, k_i of each period mim_i, with frequency λij=2πj/mi\lambda_{ij} = 2 \pi j / m_i, is two states v1(ij)v^{(ij)}_1 and v2(ij)v^{(ij)}_2 with lags 1 and 2 in the ARIMA slot of ADAM's core, uij,t=v1,t−1(ij)+v2,t−2(ij),v1,t(ij)=2cos⁡(λij)uij,t+γ1iϵt,v2,t(ij)=−uij,t+(sin⁡(λij)γ2i−cos⁡(λij)γ1i)ϵt,\begin{aligned} u_{ij,t} &= v^{(ij)}_{1,t-1} + v^{(ij)}_{2,t-2}, \\ v^{(ij)}_{1,t} &= 2 \cos(\lambda_{ij}) \, u_{ij,t} + \gamma_{1i} \epsilon_t, \\ v^{(ij)}_{2,t} &= -u_{ij,t} + \left(\sin(\lambda_{ij}) \gamma_{2i} - \cos(\lambda_{ij}) \gamma_{1i}\right) \epsilon_t, \end{aligned} so that, with the backshift operator BB, (1−2cos⁡(λij)B+B2)uij,t=(γ1iB+(sin⁡(λij)γ2i−cos⁡(λij)γ1i)B2)ϵt.(1 - 2 \cos(\lambda_{ij}) B + B^2) \, u_{ij,t} = \left(\gamma_{1i} B + (\sin(\lambda_{ij}) \gamma_{2i} - \cos(\lambda_{ij}) \gamma_{1i}) B^2\right) \epsilon_t. The measurement equation is yt(λ)=lt−1+ϕbt−1+∑i∑j=1kiuij,t+at+ϵt,y_t^{(\lambda)} = l_{t-1} + \phi b_{t-1} + \sum_i \sum_{j=1}^{k_i} u_{ij,t} + a_t + \epsilon_t, where ata_t is an optional ARMA component of ADAM: an additive block of states whose output enters the measurement equation, not ARMA errors. λ∈[0,1]\lambda \in [0, 1] is estimated by likelihood with the Jacobian of the transform. The harmonics are preselected by information criterion on a global regression of the transformed series on Fourier terms, and the trend and the ARMA orders are selected by AICc. With the default distribution="auto", the structure is selected with the generalised normal distribution of ϵt\epsilon_t, and the closest of the normal, Laplace and S distributions to its estimated shape is then fitted on that structure. Code: R/adam-tbats.R and python/src/smooth/adam_general/core/tbats/ of the smooth repository, at the code versions the evidence names. D1
  • SupportS 94smooth TBATS without ARMA is the model of De Livera TBATS: with the same smoothing parameters, damping, Box-Cox parameter and initial states, its one-step fitted values in the Box-Cox space differ from those of an independent implementation of De Livera's recursion by at most 5.1e-14 in relative terms over the 360 cases of the formulation checks. C1
  • SupportS 100For a local level with a stationary ARMA part, the additive ARMA component of smooth TBATS keeps the permanent effect of a shock in the level and its transitory effect in the ARMA part, unlike the ARMA errors of De Livera TBATS that pass the persistence of the ARMA part into the level. C21
  • SupportS 90On M4 hourly with fixed-origin evaluation, in the Nemenyi test at level 0.99 on the per-series RMSSE of the five methods, the mean rank of smooth TBATS is lower than those of forecast::tbats and AutoTBATS by more than the critical distance of 0.3577. C13
  • SupportS 90On electricity hourly with rolling-origin evaluation, in the Nemenyi test at level 0.99 on the per-series RMSSE of the five methods, the mean rank of smooth TBATS differs from those of forecast::tbats and AutoTBATS by less than the critical distance of 0.3789. C17

Context

Extends De Livera et al. (2011) · from C1
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Contradicts De Livera et al. (2011) · from C3
De Livera, Hyndman and Snyder's TBATS has ARMA errors that drive every state; the ARMA of smooth's TBATS is an additive component of ADAM, so the two models differ whenever ARMA orders are used.
Extends De Livera et al. (2011) · from C4
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C5
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C6
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C7
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C8
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C9
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C10
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C11
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C12
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C13
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C14
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C15
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C16
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C17
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C18
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C19
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends De Livera et al. (2011) · from C20
De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations.
Extends Svetunkov (2023) · from C21
The book defines ADAM's ARIMA as a component added to the other states in a single-source-of-error model.
Extends De Livera et al. (2011) · from C23
De Livera, Hyndman and Snyder's TBATS models the ARMA part as the errors that drive every state, including the level.
Extends Svetunkov (2023) · from C24
The book defines ADAM's ARIMA as a component added to the other states in a single-source-of-error model.

Definitions

Methods 7

D1 smooth TBATS: TBATS of the smooth package (R tbats(), Python TBATS): a linear single-source-of-error state space model of the Box-Cox transform yt(λ)y_t^{(\lambda)} of the series, built on ADAM's state space core, yt(λ)=w⊤vt−𝒍+ϵt,vt=Fvt−𝒍+gϵt,\begin{aligned} y_t^{(\lambda)} &= w^\top v_{t-\boldsymbol{l}} + \epsilon_t, \\ v_t &= F v_{t-\boldsymbol{l}} + g \epsilon_t, \end{aligned} where vt−𝒍v_{t-\boldsymbol{l}} takes each state at its own lag. The level ltl_t and the trend btb_t are those of ETS(A,N,N), ETS(A,A,N) or ETS(A,Ad,N), lt=lt−1+ϕbt−1+αϵt,bt=ϕbt−1+βϵt.\begin{aligned} l_t &= l_{t-1} + \phi b_{t-1} + \alpha \epsilon_t, \\ b_t &= \phi b_{t-1} + \beta \epsilon_t. \end{aligned} Each harmonic j=1,…,kij = 1, \dots, k_i of each period mim_i, with frequency λij=2πj/mi\lambda_{ij} = 2 \pi j / m_i, is two states v1(ij)v^{(ij)}_1 and v2(ij)v^{(ij)}_2 with lags 1 and 2 in the ARIMA slot of ADAM's core, uij,t=v1,t−1(ij)+v2,t−2(ij),v1,t(ij)=2cos⁡(λij)uij,t+γ1iϵt,v2,t(ij)=−uij,t+(sin⁡(λij)γ2i−cos⁡(λij)γ1i)ϵt,\begin{aligned} u_{ij,t} &= v^{(ij)}_{1,t-1} + v^{(ij)}_{2,t-2}, \\ v^{(ij)}_{1,t} &= 2 \cos(\lambda_{ij}) \, u_{ij,t} + \gamma_{1i} \epsilon_t, \\ v^{(ij)}_{2,t} &= -u_{ij,t} + \left(\sin(\lambda_{ij}) \gamma_{2i} - \cos(\lambda_{ij}) \gamma_{1i}\right) \epsilon_t, \end{aligned} so that, with the backshift operator BB, (1−2cos⁡(λij)B+B2)uij,t=(γ1iB+(sin⁡(λij)γ2i−cos⁡(λij)γ1i)B2)ϵt.(1 - 2 \cos(\lambda_{ij}) B + B^2) \, u_{ij,t} = \left(\gamma_{1i} B + (\sin(\lambda_{ij}) \gamma_{2i} - \cos(\lambda_{ij}) \gamma_{1i}) B^2\right) \epsilon_t. The measurement equation is yt(λ)=lt−1+ϕbt−1+∑i∑j=1kiuij,t+at+ϵt,y_t^{(\lambda)} = l_{t-1} + \phi b_{t-1} + \sum_i \sum_{j=1}^{k_i} u_{ij,t} + a_t + \epsilon_t, where ata_t is an optional ARMA component of ADAM: an additive block of states whose output enters the measurement equation, not ARMA errors. λ∈[0,1]\lambda \in [0, 1] is estimated by likelihood with the Jacobian of the transform. The harmonics are preselected by information criterion on a global regression of the transformed series on Fourier terms, and the trend and the ARMA orders are selected by AICc. With the default distribution="auto", the structure is selected with the generalised normal distribution of ϵt\epsilon_t, and the closest of the normal, Laplace and S distributions to its estimated shape is then fitted on that structure. Code: R/adam-tbats.R and python/src/smooth/adam_general/core/tbats/ of the smooth repository, at the code versions the evidence names.

D20 · a kind of D1 smooth TBATS dnorm: smooth TBATS with the normal distribution of ϵt\epsilon_t: TBATS(lags=[1, 24, 168], distribution="dnorm") of the Python smooth package, every other setting at its default. The structure is selected and fitted with the normal distribution only, instead of the generalised normal distribution followed by the closest named distribution of the default distribution="auto".

D2 De Livera TBATS: TBATS of De Livera, Hyndman and Snyder (2011): for the Box-Cox transform yt(λ)y_t^{(\lambda)}, yt(λ)=lt−1+ϕbt−1+∑i∑j=1kisj,t−1(i)+dt,lt=lt−1+ϕbt−1+αdt,bt=ϕbt−1+βdt,sj,t(i)=cos⁡(λij)sj,t−1(i)+sin⁡(λij)sj,t−1*(i)+γ1idt,sj,t*(i)=−sin⁡(λij)sj,t−1(i)+cos⁡(λij)sj,t−1*(i)+γ2idt,dt=∑k=1pφkdt−k+∑k=1qθkϵt−k+ϵt,\begin{aligned} y_t^{(\lambda)} &= l_{t-1} + \phi b_{t-1} + \sum_i \sum_{j=1}^{k_i} s^{(i)}_{j,t-1} + d_t, \\ l_t &= l_{t-1} + \phi b_{t-1} + \alpha d_t, \\ b_t &= \phi b_{t-1} + \beta d_t, \\ s^{(i)}_{j,t} &= \cos(\lambda_{ij}) s^{(i)}_{j,t-1} + \sin(\lambda_{ij}) s^{*(i)}_{j,t-1} + \gamma_{1i} d_t, \\ s^{*(i)}_{j,t} &= -\sin(\lambda_{ij}) s^{(i)}_{j,t-1} + \cos(\lambda_{ij}) s^{*(i)}_{j,t-1} + \gamma_{2i} d_t, \\ d_t &= \sum_{k=1}^{p} \varphi_k d_{t-k} + \sum_{k=1}^{q} \theta_k \epsilon_{t-k} + \epsilon_t, \end{aligned} with λij=2πj/mi\lambda_{ij} = 2 \pi j / m_i: ARMA(p, q) errors dtd_t drive every state. Source: De Livera et al. (2011).

D3 forecast::tbats: The tbats() function of the R package forecast (Hyndman et al.), De Livera et al.'s TBATS with its own model selection, called with seasonal.periods = c(24, 168), single-threaded, otherwise default settings. Source: Hyndman and Khandakar (2008).

D4 AutoTBATS: AutoTBATS of the Python package statsforecast (Nixtla), called with season_length = [24, 168], otherwise default settings. Source: https://github.com/Nixtla/statsforecast.

D5 ADAM ETS: ADAM ETS of smooth's Python port, ADAM(model='ZXZ', lags=[24, 168]) with its default settings: ETS with a daily and a weekly seasonal component selected by AICc. Source: Svetunkov (2023).

D6 seasonal naive 168: Seasonal naive forecast repeating the last 168 observations (the last week) of the training sample.

Data 3

D7 M4 hourly: The 414 hourly series of the M4 competition (Makridakis et al., 2020), training samples of 700 or 960 observations and test samples of 48, from the Mcompetitions/M4-methods repository (Dataset/Train/Hourly-train.csv, Dataset/Test/Hourly-test.csv). Source: Makridakis et al. (2020).

D8 M4 hourly validation split: The M4 hourly training samples with their last 48 observations held out as a validation test: fitting on observations 1..n-48 and evaluating on n-47..n; never the official test samples.

D18 electricity hourly: The 370 hourly series of electricity consumption (kW) of clients of a Portuguese utility in GluonTS's electricity_nips archive (SR3), drawn from UCI ElectricityLoadDiagrams20112014: each series is the archive's training series extended by its last test window, ending 2014-09-08 01:00, 1249 to 6001 hours long. Source: doi:10.24432/C58C86.

Variables and measures 6

D10 MASE: Mean absolute scaled error of a series over the horizon h=48h = 48, MASE=1h∑j=1h|yn+j−y^n+j|1n−24∑t=25n|yt−yt−24|,\mathrm{MASE} = \frac{\frac{1}{h} \sum_{j=1}^{h} |y_{n+j} - \hat y_{n+j}|}{\frac{1}{n-24} \sum_{t=25}^{n} |y_t - y_{t-24}|}, scaled by the in-sample mean absolute seasonal difference at lag 24, as in M4. Source: Hyndman and Koehler (2006).

D11 relative MASE: Geometric mean over series of the ratio of the MASE of method A to that of method B; below 1, A is more accurate. Its 99% confidence interval is the exponential of the t-interval of the mean log ratio over series.

D12 coverage: Share of the test values of a set of series and horizons that lie inside the 95% prediction intervals of a method.

D13 elapsed time: Wall-clock seconds to fit a method to one training sample and produce its forecasts and intervals, single-threaded.

D16 RMSSE: Root mean squared scaled error of a series over the horizon h=48h = 48, RMSSE=1h∑j=1h(yn+j−y^n+j)21n−1∑t=2n(yt−yt−1)2,\mathrm{RMSSE} = \sqrt{\frac{\frac{1}{h} \sum_{j=1}^{h} (y_{n+j} - \hat y_{n+j})^2}{\frac{1}{n-1} \sum_{t=2}^{n} (y_t - y_{t-1})^2}}, scaled by the in-sample root mean squared first difference of the fitting sample, as in M5. Source: Makridakis et al. (2022).

D17 relative RMSSE: Geometric mean over series of the ratio of the RMSSE of method A to that of method B; below 1, A is more accurate. Its 99% confidence interval is the exponential of the t-interval of the mean log ratio over series.

Study design 4

D9 fixed-origin evaluation: Each method is fitted once to each training sample and forecasts h = 1..48 steps ahead from its end; no refits, no rolling origin.

D14 formulation checks: Deterministic checks of smooth TBATS against its specification, per seed: (H1) random parameters, initial states and series of 300 observations simulated by De Livera's recursion with normal errors, for the periods {24}, {7.3}, {24, 168}, {7.3, 30.4}, no/additive/damped trend and lambda in {1, 0.5, 0}, fitted by smooth TBATS with all parameters and initial states fixed; (H2) a noise-free series of 300 observations made of a random level and random harmonics of the periods 7.3 and 24, fitted with random fixed persistence and initial='backcasting'; (H3) nine random pairs of alpha in (0.05, 0.95) and |phi| in (0.05, 0.95) for a local level with AR(1), the impulse response w'F^(j-1)g at j = 1..20 read from the fitted model's matrices. Code: claimset-code/smooth-tbats/formulation_checks.py.

D15 Nemenyi test: The Nemenyi test as computed by greybox's rmcb (distribution 'tukey', ties averaged): the methods are ranked within each series by the error measure, and two methods differ at level 1−α1 - \alpha when their mean ranks over series differ by more than the critical distance CD=q1−α(k,∞)2k(k+1)6N\mathrm{CD} = \frac{q_{1-\alpha}(k, \infty)}{\sqrt{2}} \sqrt{\frac{k (k + 1)}{6 N}} for kk methods and NN series. Source: https://jmlr.org/papers/v7/demsar06a.html.

D19 rolling-origin evaluation: Seven daily forecast origins over the last week of each series: the last one 48 hours before its end, the others 24, 48, ..., 144 hours earlier. At each origin each method is fitted to the 1008 preceding hours (six weeks) and forecasts 48 hours ahead; the RMSSE and the interval coverage of a series are averaged over its seven origins.

Assumptions

C22 · theorem_condition · untestable

The autoregressive polynomial ϕ(B)\phi(B) of the ARMA part has all its roots outside the unit circle, and its moving average polynomial θ(B)\theta(B) is invertible. Rested on by C21, C23, C24, E6.

Principal claims

SupportS 94C1 · methodological · planned test · novelty: new, by the authors

smooth TBATS without ARMA is the model of De Livera TBATS: with the same smoothing parameters, damping, Box-Cox parameter and initial states, its one-step fitted values in the Box-Cox space differ from those of an independent implementation of De Livera's recursion by at most 5.1e-14 in relative terms over the 360 cases of the formulation checks.

Scope: data: simulated series; conditions: seeds 101 to 110, series of 300 observations, periods 24, 7.3, 168 and 30.4, Box-Cox parameter 1, 0.5 and 0, no, additive and damped trend · Simulation: E1 (R1), artefacts A10 · Extends: De Livera et al. (2011) · Planned: lab plan PL1, planned 11 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 94C3 · methodological · planned test · novelty: new, by the authors

The ARMA component of smooth TBATS is ADAM's additive ARMA component, not the ARMA errors of De Livera TBATS: for a local level with an AR(1) component, its impulse response at lags 1 to 20 differs from that of the additive component by at most 2.2e-16 and from that of ARMA errors by at least 0.01651 in each of nine pairs of smoothing parameter and AR coefficient.

Scope: data: fitted models with fixed parameters; conditions: a local level with an AR(1) component, nine random pairs of smoothing parameter in (0.05, 0.95) and |AR coefficient| in (0.05, 0.95), impulse response at lags 1 to 20 · Simulation: E1 (R3, R4), artefacts A10 · Contradicts: De Livera et al. (2011) · Planned: lab plan PL1, planned 11 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 89C10 · descriptive · planned test · novelty: new, by the authors

On M4 hourly with fixed-origin evaluation, smooth TBATS is not less accurate than forecast::tbats in RMSSE beyond the non-inferiority margin of 1.05: the relative RMSSE of smooth TBATS to forecast::tbats is 0.848 (99% confidence interval 0.7969 to 0.9024).

Scope: data: the 414 hourly series of the M4 competition; conditions: one fixed origin per series: the M4 test split, horizon 48, periods 24 and 168, smooth (Python port) at the commit of the run, comparator: forecast::tbats, measure: RMSSE · Data: E3 (R1, T3.r1), artefacts A8 · Extends: De Livera et al. (2011) · Planned: lab plan PL5, planned 12 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 90C13 · descriptive · planned test · novelty: new, by the authors

On M4 hourly with fixed-origin evaluation, in the Nemenyi test at level 0.99 on the per-series RMSSE of the five methods, the mean rank of smooth TBATS is lower than those of forecast::tbats and AutoTBATS by more than the critical distance of 0.3577.

Scope: data: the 414 hourly series of the M4 competition; conditions: one fixed origin per series: the M4 test split, horizon 48, periods 24 and 168, smooth (Python port) at the commit of the run, the five methods ranked within each series · Data: E3 (T1.r1, T1.r2, T1.r3, R6), artefacts A8 · Extends: De Livera et al. (2011) · Planned: lab plan PL5, planned 12 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 90C17 · descriptive · planned test · novelty: new, by the authors

On electricity hourly with rolling-origin evaluation, in the Nemenyi test at level 0.99 on the per-series RMSSE of the five methods, the mean rank of smooth TBATS differs from those of forecast::tbats and AutoTBATS by less than the critical distance of 0.3789.

Scope: data: 370 hourly electricity series of clients of one Portuguese utility, 2011-2014; conditions: 7 daily origins in the last week of each series, ending 2014-09-08, 1008-hour fitting windows, horizon 48, periods 24 and 168, the five methods ranked within each series · Data: E4 (T1.r1, T1.r2, T1.r3, R6), artefacts A8 · Extends: De Livera et al. (2011) · Qualifies: C13 · Planned: lab plan PL7, planned 12 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 100C21 · formal · novelty: new, by the authors

For a local level with a stationary ARMA part, the additive ARMA component of smooth TBATS keeps the permanent effect of a shock in the level and its transitory effect in the ARMA part, unlike the ARMA errors of De Livera TBATS that pass the persistence of the ARMA part into the level.

Scope: conditions: a single source of error, a local level without trend or seasonal states, an ARMA part with stationary autoregressive polynomial, both formulations compared · Proof: E6 · Uses: C23, C24 · Assumes: C22 · Explains: C3 · Extends: Svetunkov (2023) · Verification: checked by a person

Supporting claims

SupportS 97C2 · methodological · planned test

Backcasting the initial states through the harmonic block of smooth TBATS is exact: on noise-free series made of a level and harmonics, the largest absolute in-sample error over the 10 series of the formulation checks is 1.8e-9.

Scope: data: simulated noise-free series; conditions: seeds 101 to 110, series of 300 observations, a level and harmonics of the periods 7.3 and 24 · Simulation: E1 (R2), artefacts A10 · Planned: lab plan PL1, planned 11 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 93C4 · descriptive · planned test · novelty: new, by the authors

On M4 hourly with fixed-origin evaluation, smooth TBATS is not less accurate than forecast::tbats in MASE beyond the non-inferiority margin of 1.05: the relative MASE of smooth TBATS to forecast::tbats is 0.8318 (99% confidence interval 0.7793 to 0.8877).

Scope: data: the 414 hourly series of the M4 competition; conditions: one fixed origin per series: the M4 test split, horizon 48, periods 24 and 168, smooth (Python port) at the commit of the run, comparator: forecast::tbats, measure: MASE · Data: E2 (R1, T3.r1), artefacts A8 · Extends: De Livera et al. (2011) · Supports: C10 · Planned: lab plan PL3, planned 11 November 2026 · Outcome: computed from the recorded results · Weaknesses: the held-out data was used elsewhere

SupportS 93C5 · descriptive · planned test · novelty: new, by the authors

On M4 hourly with fixed-origin evaluation, smooth TBATS is not less accurate than AutoTBATS in MASE beyond the non-inferiority margin of 1.05: the relative MASE of smooth TBATS to AutoTBATS is 0.7803 (99% confidence interval 0.7242 to 0.8407).

Scope: data: the 414 hourly series of the M4 competition; conditions: one fixed origin per series: the M4 test split, horizon 48, periods 24 and 168, smooth (Python port) at the commit of the run, comparator: AutoTBATS · Data: E2 (R2, T3.r2), artefacts A8 · Extends: De Livera et al. (2011) · Supports: C11 · Planned: lab plan PL3, planned 11 November 2026 · Outcome: computed from the recorded results · Weaknesses: the held-out data was used elsewhere

SupportS 93C6 · descriptive · planned test · novelty: new, by the authors

On M4 hourly with fixed-origin evaluation, smooth TBATS is more accurate in MASE than seasonal naive 168: the relative MASE of smooth TBATS to seasonal naive 168 is 0.5067 (99% confidence interval 0.4479 to 0.5734).

Scope: data: the 414 hourly series of the M4 competition; conditions: one fixed origin per series: the M4 test split, horizon 48, periods 24 and 168, smooth (Python port) at the commit of the run, comparator: seasonal naive 168 · Data: E2 (R4, T3.r3), artefacts A8 · Extends: De Livera et al. (2011) · Supports: C12 · Planned: lab plan PL3, planned 11 November 2026 · Outcome: computed from the recorded results · Weaknesses: the held-out data was used elsewhere

SupportS 93C7 · descriptive · planned test · novelty: new, by the authors

On M4 hourly, smooth TBATS takes less elapsed time per series than forecast::tbats: the geometric mean over series of the ratio of their elapsed times is 0.5791 (99% confidence interval 0.539 to 0.6222).

Scope: data: the 414 hourly series of the M4 competition; conditions: one fixed origin per series: the M4 test split, horizon 48, periods 24 and 168, smooth (Python port) at the commit of the run, comparator: forecast::tbats, measure: elapsed time · Data: E2 (R5, T3.r4), artefacts A8 · Extends: De Livera et al. (2011) · Planned: lab plan PL3, planned 11 November 2026 · Outcome: computed from the recorded results · Weaknesses: the held-out data was used elsewhere

SupportS 100C8 · descriptive · planned test · novelty: new, by the authors

On M4 hourly, the 95% prediction intervals of smooth TBATS have a coverage of 0.9273, above the threshold of 0.9 the plan set.

Scope: data: the 414 hourly series of the M4 competition; conditions: one fixed origin per series: the M4 test split, horizon 48, periods 24 and 168, smooth (Python port) at the commit of the run, nominal level 0.95 · Data: E2 (T1.r1), artefacts A8 · Extends: De Livera et al. (2011) · Planned: lab plan PL3, planned 11 November 2026 · Outcome: computed from the recorded results · Weaknesses: the held-out data was used elsewhere

SupportS 94C9 · descriptive · planned test · novelty: new, by the authors

On M4 hourly with fixed-origin evaluation, in the Nemenyi test at level 0.99 on the per-series MASE of the five methods, the mean rank of smooth TBATS is lower than those of forecast::tbats and AutoTBATS by more than the critical distance of 0.3577.

Scope: data: the 414 hourly series of the M4 competition; conditions: one fixed origin per series: the M4 test split, horizon 48, periods 24 and 168, smooth (Python port) at the commit of the run, the five methods ranked within each series · Data: E2 (T1.r1, T1.r2, T1.r3, R6), artefacts A8 · Extends: De Livera et al. (2011) · Supports: C13 · Planned: lab plan PL3, planned 11 November 2026 · Outcome: computed from the recorded results · Weaknesses: the held-out data was used elsewhere

SupportS 92C11 · descriptive · planned test · novelty: new, by the authors

On M4 hourly with fixed-origin evaluation, smooth TBATS is not less accurate than AutoTBATS in RMSSE beyond the non-inferiority margin of 1.05: the relative RMSSE of smooth TBATS to AutoTBATS is 0.7959 (99% confidence interval 0.7411 to 0.8549).

Scope: data: the 414 hourly series of the M4 competition; conditions: one fixed origin per series: the M4 test split, horizon 48, periods 24 and 168, smooth (Python port) at the commit of the run, comparator: AutoTBATS · Data: E3 (R2, T3.r2), artefacts A8 · Extends: De Livera et al. (2011) · Supports: C13 · Planned: lab plan PL5, planned 12 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 92C12 · descriptive · planned test · novelty: new, by the authors

On M4 hourly with fixed-origin evaluation, smooth TBATS is more accurate in RMSSE than seasonal naive 168: the relative RMSSE of smooth TBATS to seasonal naive 168 is 0.5358 (99% confidence interval 0.4785 to 0.6001).

Scope: data: the 414 hourly series of the M4 competition; conditions: one fixed origin per series: the M4 test split, horizon 48, periods 24 and 168, smooth (Python port) at the commit of the run, comparator: seasonal naive 168 · Data: E3 (R4, T3.r3), artefacts A8 · Extends: De Livera et al. (2011) · Planned: lab plan PL5, planned 12 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 92C14 · descriptive · planned test · novelty: new, by the authors

On electricity hourly with rolling-origin evaluation, smooth TBATS is not shown to be more accurate than forecast::tbats at level 0.99: the relative RMSSE of smooth TBATS to forecast::tbats is 0.9753 (99% confidence interval 0.947 to 1.004).

Scope: data: 370 hourly electricity series of clients of one Portuguese utility, 2011-2014; conditions: 7 daily origins in the last week of each series, ending 2014-09-08, 1008-hour fitting windows, horizon 48, periods 24 and 168, comparator: forecast::tbats · Data: E4 (R1, T3.r1), artefacts A8 · Extends: De Livera et al. (2011) · Qualifies: C10 · Planned: lab plan PL7, planned 12 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 92C15 · descriptive · planned test · novelty: new, by the authors

On electricity hourly with rolling-origin evaluation, smooth TBATS is not shown to be more accurate than AutoTBATS at level 0.99: the relative RMSSE of smooth TBATS to AutoTBATS is 0.9909 (99% confidence interval 0.9753 to 1.007).

Scope: data: 370 hourly electricity series of clients of one Portuguese utility, 2011-2014; conditions: 7 daily origins in the last week of each series, ending 2014-09-08, 1008-hour fitting windows, horizon 48, periods 24 and 168, comparator: AutoTBATS · Data: E4 (R2, T3.r2), artefacts A8 · Extends: De Livera et al. (2011) · Qualifies: C11 · Planned: lab plan PL7, planned 12 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 92C16 · descriptive · planned test · novelty: new, by the authors

On electricity hourly with rolling-origin evaluation, smooth TBATS is more accurate than seasonal naive 168: the relative RMSSE of smooth TBATS to seasonal naive 168 is 0.8244 (99% confidence interval 0.7981 to 0.8515).

Scope: data: 370 hourly electricity series of clients of one Portuguese utility, 2011-2014; conditions: 7 daily origins in the last week of each series, ending 2014-09-08, 1008-hour fitting windows, horizon 48, periods 24 and 168, comparator: seasonal naive 168 · Data: E4 (R4, T3.r3), artefacts A8 · Extends: De Livera et al. (2011) · Supports: C12 · Planned: lab plan PL7, planned 12 November 2026 · Outcome: computed from the recorded results · Weaknesses: the code changed after the plan

SupportS 93C18 · descriptive · planned test · novelty: new, by the authors

On electricity hourly with rolling-origin evaluation, smooth TBATS dnorm is not shown to be less accurate than smooth TBATS with its default distribution: the relative RMSSE of smooth TBATS dnorm to smooth TBATS is 0.9275 (99% confidence interval 0.7825 to 1.099).

Scope: data: 370 hourly electricity series of clients of one Portuguese utility, 2011-2014; conditions: 7 daily origins in the last week of each series, ending 2014-09-08, 1008-hour fitting windows, horizon 48, periods 24 and 168, comparator: smooth TBATS · Data: E5 (R1, T3.r1), artefacts A8 · Extends: De Livera et al. (2011) · Qualifies: C19 · Planned: lab plan PL8, planned 13 November 2026 · Outcome: computed from the recorded results

SupportS 93C19 · descriptive · planned test · novelty: new, by the authors

On electricity hourly with rolling-origin evaluation, the 95% prediction intervals of smooth TBATS dnorm cover fewer test values than the nominal level: their mean coverage minus 0.95 is -0.0168 (99% confidence interval -0.02364 to -0.009956).

Scope: data: 370 hourly electricity series of clients of one Portuguese utility, 2011-2014; conditions: 7 daily origins in the last week of each series, ending 2014-09-08, 1008-hour fitting windows, horizon 48, periods 24 and 168, nominal level 0.95 · Data: E5 (R3, T1.r2, T3.r2), artefacts A8 · Extends: De Livera et al. (2011) · Qualifies: C20 · Planned: lab plan PL8, planned 13 November 2026 · Outcome: computed from the recorded results

SupportS 90C20 · descriptive · planned test · novelty: new, by the authors

On electricity hourly with rolling-origin evaluation, smooth TBATS dnorm takes less elapsed time per series than smooth TBATS with its default distribution: the geometric mean over series of the ratio of their elapsed times is 0.6368 (99% confidence interval 0.619 to 0.6552).

Scope: data: 370 hourly electricity series of clients of one Portuguese utility, 2011-2014; conditions: 7 daily origins in the last week of each series, ending 2014-09-08, 1008-hour fitting windows, horizon 48, periods 24 and 168, comparator: smooth TBATS · Data: E5 (R2, T3.r3), artefacts A8 · Extends: De Livera et al. (2011) · Planned: lab plan PL8, planned 13 November 2026 · Outcome: computed from the recorded results

SupportS 100C23 · formal · novelty: new, by the authors

Under the ARMA errors of De Livera TBATS, for a local level with smoothing parameter α>0\alpha > 0 and an ARMA part with polynomials ϕ(B)\phi(B) and θ(B)\theta(B), a unit shock moves the level permanently by αθ(1)/ϕ(1)\alpha \theta(1) / \phi(1), unbounded as a root of ϕ(B)\phi(B) approaches 1.

Scope: conditions: a single source of error, a local level without trend or seasonal states, an ARMA part with stationary autoregressive polynomial, ARMA errors · Proof: E6 · Assumes: C22 · Extends: De Livera et al. (2011) · Verification: checked by a person · Used by: C21

SupportS 100C24 · formal · novelty: new, by the authors

Under the additive ARMA component of smooth TBATS, for a local level with smoothing parameter α\alpha and a stationary ARMA part, a unit shock moves the level permanently by α\alpha and the response of the ARMA part decays to zero.

Scope: conditions: a single source of error, a local level without trend or seasonal states, an ARMA part with stationary autoregressive polynomial, an additive ARMA component · Proof: E6 · Assumes: C22 · Extends: Svetunkov (2023) · Verification: checked by a person · Used by: C21

Evidence

E1 · simulation · cited by C1, C2, C3

The formulation checks of smooth TBATS, run once with the seeds 101 to 110 (lab run RN6, plan PL1): the largest relative difference of the one-step fitted values from an independent implementation of the recursion of De Livera TBATS over 360 cases (H1), the largest absolute in-sample error of backcasting on 10 noise-free series (H2), and, over nine pairs of smoothing parameter and AR(1) coefficient, the largest distance of the impulse response of smooth TBATS from that of an additive ARMA component and the smallest from that of ARMA errors (H3). The per-case values are in the CSV of the lab results (A8).

Setup: studied smooth TBATS, De Livera TBATS · procedure formulation checks

  • R1 max relative difference of fitted values over all cases, smooth TBATS vs De Livera recursion (max) = 0.0000 (seeds: [101, 102, 103, 104, 105, 106, 107, 108, 109, 110], cases: 360)
  • R2 max absolute in-sample error over all noise-free series, backcasting (max) = 0.0000 (seeds: [101, 102, 103, 104, 105, 106, 107, 108, 109, 110], n: 300)
  • R3 max absolute difference of smooth's impulse response from alpha + phi^j over all cases (max) = 0.0000
  • R4 min over cases of the max absolute difference of smooth's impulse response from the ARMA-errors response (min) = 0.0165

E2 · quant · secondary · cited by C4, C5, C6, C7, C8, C9

Benchmark of smooth TBATS against forecast::tbats, AutoTBATS, ADAM ETS and seasonal naive 168 on the 414 series of M4 hourly with fixed-origin evaluation (the M4 test split, 48 hours ahead), run once under plan PL3 (lab run RN8). Each fit is single-threaded; a failed fit gets the seasonal naive 168 forecasts and no interval. Per series, MASE, the coverage of the 95% interval and the elapsed time (averaged over the origins where there are several); the relative MASE of smooth TBATS to each comparator is the geometric mean over series of the ratio of their MASE, with a 99% t-interval on the log scale; the one-sided t-tests on the mean log ratios are adjusted by Holm over four; the Nemenyi test ranks the five methods within each series. The sMAPE that the run also computed is in the CSV of the lab results only (decision DC4 of the lab record).

Setup: studied smooth TBATS, forecast::tbats, AutoTBATS, ADAM ETS, seasonal naive 168 · cases M4 hourly · procedure fixed-origin evaluation, Nemenyi test · lens MASE, relative MASE, coverage, elapsed time

  • R1 relative MASE (geometric mean) = 0.8318 99% confidence interval 0.7793 to 0.8877 (comparator: forecast::tbats)
  • R2 relative MASE (geometric mean) = 0.7803 99% confidence interval 0.7242 to 0.8407 (comparator: AutoTBATS)
  • R3 relative MASE (geometric mean) = 0.9261 99% confidence interval 0.8243 to 1.0404 (comparator: ADAM ETS)
  • R4 relative MASE (geometric mean) = 0.5067 99% confidence interval 0.4479 to 0.5734 (comparator: seasonal naive 168)
  • R5 relative elapsed time (geometric mean) = 0.5791 99% confidence interval 0.539 to 0.6222 (comparator: forecast::tbats)
  • R6 critical distance, Nemenyi test (MASE) = 0.3577 (level: 0.99)
  • R7 series in Nemenyi test (count) = 414
  • R8 p-value, rmcb test of equal mean ranks (MASE) (p_value) = 0.0000
E2.T1 · Per method: mean and median MASE over series, coverage of the 95% prediction intervals, median elapsed time per series in seconds, series with a failed fit, and the mean rank of MASE in the Nemenyi test with the bounds of its 99% interval.
RowmethodMASE (mean)MASE (median)coverage (nominal level: 0.95) (mean)elapsed time (median)series failed (count)mean rank of MASE, Nemenyi test (mean)mean rank of MASE, Nemenyi test (bound: 99% interval, lower)mean rank of MASE, Nemenyi test (bound: 99% interval, upper)
r1smooth TBATS1.0290.870.927314.1802.57252.3942.751
r2forecast::tbats1.3040.99410.910124.402.95412.7753.133
r3AutoTBATS1.5160.98360.918321.4203.07732.8983.256
r4ADAM ETS2.110.79840.90541.3102.6182.442.797
r5seasonal naive 1682.8671.3970.000003.7783.5993.957
E2.T2 · Per comparator: the series the relative MASE of smooth TBATS to it is computed on, and those left out because the MASE of smooth TBATS or of the comparator is zero. (method: smooth TBATS)
Rowcomparatorseries in relative MASE (count)series excluded by zero MASE (count)
r1forecast::tbats4140
r2AutoTBATS4140
r3ADAM ETS4140
r4seasonal naive 1684140
E2.T3 · The one-sided t-tests of the plan on the per-series log ratios of smooth TBATS to the comparator: p-value and p-value adjusted by Holm over the four tests.
Rowcomparatormeasurep-value, one-sided t-test (p_value)Holm-adjusted p-value, one-sided t-test (p_value)
r1forecast::tbatsMASE0.00000.0000
r2AutoTBATSMASE0.00000.0000
r3seasonal naive 168MASE0.00000.0000
r4forecast::tbatselapsed time0.00000.0000

E3 · quant · secondary · cited by C10, C11, C12, C13

Benchmark of smooth TBATS against forecast::tbats, AutoTBATS, ADAM ETS and seasonal naive 168 on the 414 series of M4 hourly with fixed-origin evaluation (the M4 test split, 48 hours ahead), run once under plan PL5 (lab run RN11). Each fit is single-threaded; a failed fit gets the seasonal naive 168 forecasts and no interval. Per series, RMSSE, the coverage of the 95% interval and the elapsed time (averaged over the origins where there are several); the relative RMSSE of smooth TBATS to each comparator is the geometric mean over series of the ratio of their RMSSE, with a 99% t-interval on the log scale; the one-sided t-tests on the mean log ratios are adjusted by Holm over four; the Nemenyi test ranks the five methods within each series.

Setup: studied smooth TBATS, forecast::tbats, AutoTBATS, ADAM ETS, seasonal naive 168 · cases M4 hourly · procedure fixed-origin evaluation, Nemenyi test · lens RMSSE, relative RMSSE, coverage, elapsed time

  • R1 relative RMSSE (geometric mean) = 0.848 99% confidence interval 0.7969 to 0.9024 (comparator: forecast::tbats)
  • R2 relative RMSSE (geometric mean) = 0.7959 99% confidence interval 0.7411 to 0.8549 (comparator: AutoTBATS)
  • R3 relative RMSSE (geometric mean) = 0.9483 99% confidence interval 0.847 to 1.0616 (comparator: ADAM ETS)
  • R4 relative RMSSE (geometric mean) = 0.5358 99% confidence interval 0.4785 to 0.6001 (comparator: seasonal naive 168)
  • R5 relative elapsed time (geometric mean) = 0.5682 99% confidence interval 0.5281 to 0.6113 (comparator: forecast::tbats)
  • R6 critical distance, Nemenyi test (RMSSE) = 0.3577 (level: 0.99)
  • R7 series in Nemenyi test (count) = 414
  • R8 p-value, rmcb test of equal mean ranks (RMSSE) (p_value) = 0.0000
E3.T1 · Per method: mean and median RMSSE over series, coverage of the 95% prediction intervals, median elapsed time per series in seconds, series with a failed fit, and the mean rank of RMSSE in the Nemenyi test with the bounds of its 99% interval.
RowmethodRMSSE (mean)RMSSE (median)coverage (nominal level: 0.95) (mean)elapsed time (median)series failed (count)mean rank of RMSSE, Nemenyi test (mean)mean rank of RMSSE, Nemenyi test (bound: 99% interval, lower)mean rank of RMSSE, Nemenyi test (bound: 99% interval, upper)
r1smooth TBATS0.82940.62420.927312.2502.56282.3842.742
r2forecast::tbats0.90940.6980.910121.8402.93722.7583.116
r3AutoTBATS1.0630.68540.918319.703.04352.8653.222
r4ADAM ETS2.10.63140.90551.11202.6452.4662.824
r5seasonal naive 1681.4531.0950.000003.8123.6333.99
E3.T2 · Per comparator: the series the relative RMSSE of smooth TBATS to it is computed on, and those left out because the RMSSE of smooth TBATS or of the comparator is zero. (method: smooth TBATS)
Rowcomparatorseries in relative RMSSE (count)series excluded by zero RMSSE (count)
r1forecast::tbats4140
r2AutoTBATS4140
r3ADAM ETS4140
r4seasonal naive 1684140
E3.T3 · The one-sided t-tests of the plan on the per-series log ratios of smooth TBATS to the comparator: p-value and p-value adjusted by Holm over the four tests.
Rowcomparatormeasurep-value, one-sided t-test (p_value)Holm-adjusted p-value, one-sided t-test (p_value)
r1forecast::tbatsRMSSE0.00000.0000
r2AutoTBATSRMSSE0.00000.0000
r3seasonal naive 168RMSSE0.00000.0000
r4forecast::tbatselapsed time0.00000.0000

E4 · quant · secondary · cited by C14, C15, C16, C17

Benchmark of smooth TBATS against forecast::tbats, AutoTBATS, ADAM ETS and seasonal naive 168 on the 370 series of electricity hourly with rolling-origin evaluation (7 daily origins of the last week, 1008-hour fitting windows, 48 hours ahead; series 322, constant in its windows, has an undefined RMSSE and is left out), run once under plan PL7 (lab run RN13). Each fit is single-threaded; a failed fit gets the seasonal naive 168 forecasts and no interval. Per series, RMSSE, the coverage of the 95% interval and the elapsed time (averaged over the origins where there are several); the relative RMSSE of smooth TBATS to each comparator is the geometric mean over series of the ratio of their RMSSE, with a 99% t-interval on the log scale; the one-sided t-tests on the mean log ratios are adjusted by Holm over four; the Nemenyi test ranks the five methods within each series.

Setup: studied smooth TBATS, forecast::tbats, AutoTBATS, ADAM ETS, seasonal naive 168 · cases electricity hourly · procedure rolling-origin evaluation, Nemenyi test · lens RMSSE, relative RMSSE, coverage, elapsed time

  • R1 relative RMSSE (geometric mean) = 0.9753 99% confidence interval 0.947 to 1.0044 (comparator: forecast::tbats)
  • R2 relative RMSSE (geometric mean) = 0.9909 99% confidence interval 0.9753 to 1.0067 (comparator: AutoTBATS)
  • R3 relative RMSSE (geometric mean) = 1.0043 99% confidence interval 0.7973 to 1.265 (comparator: ADAM ETS)
  • R4 relative RMSSE (geometric mean) = 0.8244 99% confidence interval 0.7981 to 0.8515 (comparator: seasonal naive 168)
  • R5 relative elapsed time (geometric mean) = 0.6094 99% confidence interval 0.58 to 0.6404 (comparator: forecast::tbats)
  • R6 critical distance, Nemenyi test (RMSSE) = 0.3789 (level: 0.99)
  • R7 series in Nemenyi test (count) = 369
  • R8 p-value, rmcb test of equal mean ranks (RMSSE) (p_value) = 0.0000
  • R9 series excluded by undefined RMSSE (count) = 1
E4.T1 · Per method: mean and median RMSSE over series, coverage of the 95% prediction intervals, median elapsed time per series in seconds, series with a failed fit, and the mean rank of RMSSE in the Nemenyi test with the bounds of its 99% interval.
RowmethodRMSSE (mean)RMSSE (median)coverage (nominal level: 0.95) (mean)elapsed time (median)series failed (count)fits failed (count)mean rank of RMSSE, Nemenyi test (mean)mean rank of RMSSE, Nemenyi test (bound: 99% interval, lower)mean rank of RMSSE, Nemenyi test (bound: 99% interval, upper)
r1smooth TBATS0.79480.66440.95412.22002.48242.2932.672
r2forecast::tbats0.79850.68780.943320.26002.55832.3692.748
r3AutoTBATS0.79490.6710.937215.07002.38752.1982.577
r4ADAM ETS0.87950.70650.97761.245003.2763.0873.466
r5seasonal naive 1681.0030.80020.0000004.2954.1064.485
E4.T2 · Per comparator: the series the relative RMSSE of smooth TBATS to it is computed on, and those left out because the RMSSE of smooth TBATS or of the comparator is zero. (method: smooth TBATS)
Rowcomparatorseries in relative RMSSE (count)series excluded by zero RMSSE (count)
r1forecast::tbats3690
r2AutoTBATS3690
r3ADAM ETS3690
r4seasonal naive 1683681
E4.T3 · The one-sided t-tests of the plan on the per-series log ratios of smooth TBATS to the comparator: p-value and p-value adjusted by Holm over the four tests.
Rowcomparatormeasurep-value, one-sided t-test (p_value)Holm-adjusted p-value, one-sided t-test (p_value)
r1forecast::tbatsRMSSE0.01410.0281
r2AutoTBATSRMSSE0.06690.0669
r3seasonal naive 168RMSSE0.00000.0000
r4forecast::tbatselapsed time0.00000.0000

E5 · quant · secondary · cited by C18, C19, C20

smooth TBATS dnorm against smooth TBATS (distribution "auto") on the 370 series of electricity hourly with rolling-origin evaluation (the windows of run RN13), run once under plan PL8 (lab run RN14). Both are fitted to each window one after the other in the same single-threaded worker, so that their elapsed times are measured under the same load; a failed fit gets the seasonal naive 168 forecasts and no interval. Per series, the RMSSE, the coverage of the 95% interval and the elapsed time averaged over the 7 origins; one-sided t-tests over series on the mean log ratio of RMSSE (auto over dnorm), the mean coverage of dnorm minus 0.95 and the mean log ratio of elapsed time (dnorm over auto), adjusted by Holm over the three; the Nemenyi test on the six methods (A1). Series 322 has an undefined RMSSE and is left out.

Setup: studied smooth TBATS dnorm, smooth TBATS · cases electricity hourly · procedure rolling-origin evaluation · lens RMSSE, relative RMSSE, coverage, elapsed time

  • R1 relative RMSSE (geometric mean) = 0.9275 99% confidence interval 0.7825 to 1.0994 (method: smooth TBATS dnorm, comparator: smooth TBATS)
  • R2 relative elapsed time (geometric mean) = 0.6368 99% confidence interval 0.619 to 0.6552 (method: smooth TBATS dnorm, comparator: smooth TBATS)
  • R3 coverage of smooth TBATS dnorm minus 0.95 (mean) = -0.0168 99% confidence interval -0.0236 to -0.01
  • R4 log ratio of RMSSE, smooth TBATS to smooth TBATS dnorm (mean) = 0.0753 99% confidence interval -0.0947 to 0.2453
  • R5 log ratio of elapsed time, smooth TBATS dnorm to smooth TBATS (mean) = -0.4512 99% confidence interval -0.4797 to -0.4228
  • R6 critical distance, Nemenyi test (RMSSE) = 0.4633 (level: 0.99)
  • R7 series in Nemenyi test (count) = 369
  • R8 p-value, rmcb test of equal mean ranks (RMSSE) (p_value) = 0.0000
  • R9 series excluded by undefined RMSSE (count) = 1
E5.T1 · Per method: mean and median RMSSE over series, coverage of the 95% prediction intervals, median elapsed time per series in seconds, and the series and fits that failed.
RowmethodRMSSE (mean)RMSSE (median)coverage (nominal level: 0.95) (mean)elapsed time (median)series failed (count)fits failed (count)
r1smooth TBATS0.79480.66440.95412.6400
r2smooth TBATS dnorm0.78360.65540.93328.03100
E5.T2 · Nemenyi test on the per-series RMSSE of the six methods: smooth TBATS and smooth TBATS dnorm from this run, the other four from run RN13 on the same windows; mean rank with the bounds of its 99% interval.
Rowmethodmean rank of RMSSE, Nemenyi test (mean)mean rank of RMSSE, Nemenyi test (bound: 99% interval, lower)mean rank of RMSSE, Nemenyi test (bound: 99% interval, upper)
r1smooth TBATS3.032.7983.261
r2forecast::tbats3.1222.893.354
r3AutoTBATS2.9212.693.153
r4ADAM ETS3.9953.7634.226
r5seasonal naive 1685.1654.9345.397
r6smooth TBATS dnorm2.7672.5352.999
E5.T3 · The three one-sided t-tests of plan PL8 over series: p-value, p-value adjusted by Holm over the three, series in the test and series left out for a non-finite value.
Rowmeasurep-value, one-sided t-test (p_value)Holm-adjusted p-value, one-sided t-test (p_value)series in the t-test (count)series excluded by a non-finite value (count)
r1RMSSE0.87380.87383690
r2coverage0.00000.00003690
r3elapsed time0.00000.00003690
Nemenyi test on the per-series RMSSE of the six methods on the electricity data (369 series, level 0.99): mean rank of each method with its 99% interval, from…
A1 Nemenyi test on the per-series RMSSE of the six methods on the electricity data (369 series, level 0.99): mean rank of each method with its 99% interval, from run RN14 (smooth TBATS, smooth TBATS dnorm) and run RN13 (the other four).

E6 · formal · cited by C21, C23, C24

Derivation of the impulse responses of a local level with an ARMA part under ARMA errors and under ADAM's additive ARMA component, and their limits.

Foundation: linear difference equations and power series in the backshift operator $B$ · Assumes: C22

  1. Under ARMA errors, ϕ(B)dt=θ(B)ϵt\phi(B) d_t = \theta(B) \epsilon_t with ϕ(B)\phi(B) stationary gives dt=ψ(B)ϵt=∑k≥0ψkϵt−kd_t = \psi(B) \epsilon_t = \sum_{k \ge 0} \psi_k \epsilon_{t-k}, with ψ(B)=θ(B)/ϕ(B)\psi(B) = \theta(B) / \phi(B), ψ0=1\psi_0 = 1, ψk→0\psi_k \to 0 and ∑k≥0ψk=ψ(1)=θ(1)/ϕ(1)\sum_{k \ge 0} \psi_k = \psi(1) = \theta(1) / \phi(1). (definition; from D2, C22)
  2. The level of the ARMA-errors model moves by lt−lt−1=αdt=αψ(B)ϵtl_t - l_{t-1} = \alpha d_t = \alpha \psi(B) \epsilon_t, so a unit shock at tt moves lt+j−1l_{t+j-1} by α∑k=0j−1ψk\alpha \sum_{k=0}^{j-1} \psi_k and yt+j=lt+j−1+dt+jy_{t+j} = l_{t+j-1} + d_{t+j} by rjE=ψj+α∑k=0j−1ψkr^{E}_j = \psi_j + \alpha \sum_{k=0}^{j-1} \psi_k for j≥1j \ge 1. (substitution; from S1, D2)
  3. limj→∞⁡rjE=αθ(1)/ϕ(1)\lim_{j \to \infty} r^{E}_j = \alpha \, \theta(1) / \phi(1): the level keeps αψ(1)\alpha \psi(1) of every shock, and its increments αψ(B)ϵt\alpha \psi(B) \epsilon_t are autocorrelated unless ψ(B)=1\psi(B) = 1, that is, unless there is no ARMA part, given that θ(B)\theta(B) is invertible. (limit; from S1, S2)
  4. Under ADAM's additive ARMA component, yt=lt−1+at+ϵty_t = l_{t-1} + a_t + \epsilon_t with at+ϵt=ψ(B)ϵta_t + \epsilon_t = \psi(B) \epsilon_t and lt−lt−1=αϵtl_t - l_{t-1} = \alpha \epsilon_t, so a unit shock at tt moves yt+jy_{t+j} by rjA=α+ψjr^{A}_j = \alpha + \psi_j for j≥1j \ge 1. (definition; from D1)
  5. limj→∞⁡rjA=α\lim_{j \to \infty} r^{A}_j = \alpha: the level keeps α\alpha of every shock, its increments αϵt\alpha \epsilon_t are white noise, and the response of the ARMA part, ψj\psi_j, decays to zero. (limit; from S1, S4)
  6. For an AR(1), ψj=ϕj\psi_j = \phi^j: rjE=ϕj+α(1−ϕj)/(1−ϕ)→α/(1−ϕ)r^{E}_j = \phi^j + \alpha (1 - \phi^j) / (1 - \phi) \to \alpha / (1 - \phi), which grows without bound as ϕ→1\phi \to 1 when α>0\alpha > 0, while rjA=α+ϕj→αr^{A}_j = \alpha + \phi^j \to \alpha. (substitution; from S3, S5)

Limitations

L1 · analytic flexibility · could bias magnitude · affects C4, C5, C6, C9, C10, C11, C12, C13 · mitigation: none

The accuracy measure of the M4 hourly tests changed after their results were seen: MASE decided the tests of C4, C5, C6 and C9 (lab run RN8), the guarantor then judged MASE unsuitable and set RMSSE (decision DC4 of the lab record), and C10 to C13 restate those hypotheses on RMSSE and test them on the same M4 test split (run RN11), so they are not independent of C4 to C6 and C9.

L2 · model specification · reduces precision · affects C4, C5, C6, C7, C10, C11, C12, C14, C15, C16, C18, C19, C20 · mitigation: none

The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow.

L3 · sampling · could bias magnitude · affects C14, C15, C16, C17, C18, C19, C20 · mitigation: none

The electricity evaluation covers one week, the 7 daily origins before 2014-09-08, of the clients of one utility; other weeks or seasons of the year may rank the methods differently.

L4 · computational · could bias magnitude · affects C7, C20 · mitigation: none

Elapsed times depend on the machine and on the language: forecast::tbats runs in R and the other methods in Python, the M4 runs ran on a cloud machine and the electricity runs on the guarantor's computer, and each time includes its language's overhead per fit.

L5 · other · reduces precision · affects C1, C2, C3, C10, C11, C12, C13, C14, C15, C16, C17 · mitigation: none

The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies.

References

  1. https://raw.githubusercontent.com/Mcompetitions/M4-methods/master/Dataset/Train/Hourly-train.csv https://raw.githubusercontent.com/Mcompetitions/M4-methods/master/Dataset/Train/Hourly-train.csv Used by: A5
  2. https://raw.githubusercontent.com/Mcompetitions/M4-methods/master/Dataset/Test/Hourly-test.csv https://raw.githubusercontent.com/Mcompetitions/M4-methods/master/Dataset/Test/Hourly-test.csv Used by: A6
  3. https://raw.githubusercontent.com/mbohlkeschneider/gluon-ts/mv_release/datasets/electricity_nips.tar.gz https://raw.githubusercontent.com/mbohlkeschneider/gluon-ts/mv_release/datasets/electricity_nips.tar.gz Used by: A7
  4. De Livera et al. (2011) De Livera, A. M., Hyndman, R. J., & Snyder, R. D. (2011). Forecasting Time Series With Complex Seasonal Patterns Using Exponential Smoothing. Journal of the American Statistical Association, 106(496), 1513–1527. doi:10.1198/jasa.2011.tm09771 Used by: Context, D2
  5. Hyndman and Khandakar (2008) Hyndman, R. J., & Khandakar, Y. (2008). Automatic Time Series Forecasting: The forecast Package for R. Journal of Statistical Software, 27(3). doi:10.18637/jss.v027.i03 Used by: D3
  6. Makridakis et al. (2020) Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2020). The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1), 54–74. doi:10.1016/j.ijforecast.2019.04.014 Used by: D7
  7. Makridakis et al. (2022) Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2022). M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 38(4), 1346–1364. doi:10.1016/j.ijforecast.2021.11.013 Used by: D16
  8. Svetunkov (2023) Svetunkov, I. (2023). Forecasting and Analytics with the Augmented Dynamic Adaptive Model (ADAM). Chapman and Hall/CRC. doi:10.1201/9781003452652 Used by: Context, D5
  9. doi:10.24432/C58C86 doi:10.24432/C58C86 Used by: D18
  10. Hyndman and Koehler (2006) Hyndman, R. J., & Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4), 679–688. doi:10.1016/j.ijforecast.2006.03.001 Used by: D10
  11. https://jmlr.org/papers/v7/demsar06a.html https://jmlr.org/papers/v7/demsar06a.html Used by: D15
  12. https://github.com/Nixtla/statsforecast https://github.com/Nixtla/statsforecast Used by: D4

Artefacts

  • A1 · figure · data_plot Nemenyi test on the per-series RMSSE of the six methods on the electricity data (369 series, level 0.99): mean rank of each method with its 99% interval, from run RN14 (smooth TBATS, smooth TBATS dnorm) and run RN13 (the other four). nemenyi_electricity_six_methods.png
  • A2 · table CSV written by run RN14 with the per-method values of smooth TBATS and smooth TBATS dnorm (E5.T1), at full precision. dnorm_methods.csv
  • A3 · table CSV written by run RN14 with the three t-tests of plan PL8 and the relative RMSSE and elapsed time with their intervals (E5.T3, E5.R1 to E5.R5), at full precision. dnorm_tests.csv
  • A4 · table CSV written by run RN14 with the Nemenyi test of the six methods: mean ranks, intervals, critical distance (E5.T2, E5.R6 to E5.R8), at full precision. nemenyi_electricity_six_methods.csv
  • A5 · data · https://raw.githubusercontent.com/Mcompetitions/M4-methods/master/Dataset/Train/Hourly-train.csv M4 Hourly-train.csv (text/csv), saved by hand on 2026-11-11; recorded in the lab record as SR1 on 2026-11-11. venue: GitHub, Mcompetitions/M4-methods. Written by others. M4 competition hourly data, train samples, one series per row Published by a third party at the URI; the claimset gives its hash, not the file, and the file keeps its publisher's terms. Hourly-train.csv
  • A6 · data · https://raw.githubusercontent.com/Mcompetitions/M4-methods/master/Dataset/Test/Hourly-test.csv M4 Hourly-test.csv (text/csv), saved by hand on 2026-11-11; recorded in the lab record as SR2 on 2026-11-11. venue: GitHub, Mcompetitions/M4-methods. Written by others. M4 competition hourly data, test samples, one series per row Published by a third party at the URI; the claimset gives its hash, not the file, and the file keeps its publisher's terms. Hourly-test.csv
  • A7 · data · https://raw.githubusercontent.com/mbohlkeschneider/gluon-ts/mv_release/datasets/electricity_nips.tar.gz GluonTS electricity_nips data set (application/gzip), saved by hand on 2026-11-12; recorded in the lab record as SR3 on 2026-11-12. venue: GitHub, mbohlkeschneider/gluon-ts (branch mv_release). Written by others. Hourly electricity consumption (kW) of 370 clients of a Portuguese utility, from the UCI ElectricityLoadDiagrams20112014 data, as prepared for DeepAR (Salinas et al.): train/data.json ends 2014-09-01 01:00, test/data.json holds 7 daily rolling windows ending 2014-09-02 to 2014-09-08 01:00, in GluonTS JSON lines. Published by a third party at the URI; the claimset gives its hash, not the file, and the file keeps its publisher's terms. electricity_nips.tar.gz
  • A8 · data Every result of the lab record (1308 results of 14 runs) at full precision: the lab result's ID, its run, the quantity as the run's script named it, its conditions and its value (both as JSON). The evidence's tables give the values to four significant digits. smooth-tbats.results.csv
  • A9 · lab_record Lab record of the study: questions, hypotheses, plans, runs, results and decisions, as an append-only, hash-chained event log (Claimset lab record specification v0.21, begun 2026-11-11; exported with claimset 0.41.6). lab.jsonl
  • A10 · code The study's scripts: formulation_checks.py (E1), m4_hourly.py and m4_forecast_tbats.R (E2, E3), electricity_hourly.py (E4) and electricity_dnorm.py (E5), at the last commit of the study's code; each evidence names the commit it ran at. smooth-tbats

Score

Support of the paper: 93, Well supported

Scores are computed by rules from the claimset file (EntryLib rubric 0.4): they check the record, not the science, and may change as the rubric and reviews develop. The rubric. Every issue as text, for an agent preparing the next version.

The mean of its 6 principal claims; the lowest is C10 at 89. Open problems in the principal claims and what they rest on, each counted once, with the most it costs a claim:

  1. −6 Open problems Limitation L1 (could bias magnitude, not mitigated): The accuracy measure of the M4 hourly tests changed after their results were seen: MASE decided the tests of C4, C5, C6 and C9 (lab run RN8), the guarantor then judged MASE unsuitable and set RMSSE (decision DC4 of the lab record), and C10 to C13 restate those hypotheses on RMSSE and test them on the same M4 test split (run RN11), so they are not independent of C4 to C6 and C9. In C4, C5, C6, C9, C10, C11, C12, C13. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −6 Open problems Limitation L3 (could bias magnitude, not mitigated): The electricity evaluation covers one week, the 7 daily origins before 2014-09-08, of the clients of one utility; other weeks or seasons of the year may rank the methods differently. In C14, C15, C16, C17, C18, C19, C20. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −3 Plan The lab record flags the test: code changed after plan. In C1, C3, C10, C13, C17. To remove it: Rerun the test as planned, or explain the flag in a limitation.
  4. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. In C4, C5, C6, C7, C10, C11, C12, C14, C15, C16, C18, C19, C20. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  5. −3 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. In C1, C2, C3, C10, C11, C12, C13, C14, C15, C16, C17. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

Claims

C1: 94, Well supported

Novelty: new, by the authors

  1. −3 Plan The lab record flags the test: code changed after plan. To remove it: Rerun the test as planned, or explain the flag in a limitation.
  2. −3 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C2: 97, Well supported

  1. −3 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C3: 94, Well supported

Novelty: new, by the authors

  1. −3 Plan The lab record flags the test: code changed after plan. To remove it: Rerun the test as planned, or explain the flag in a limitation.
  2. −3 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C4: 93, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L1 (could bias magnitude, not mitigated): The accuracy measure of the M4 hourly tests changed after their results were seen: MASE decided the tests of C4, C5, C6 and C9 (lab run RN8), the guarantor then judged MASE unsuitable and set RMSSE (decision DC4 of the lab record), and C10 to C13 restate those hypotheses on RMSSE and test them on the same M4 test split (run RN11), so they are not independent of C4 to C6 and C9. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C5: 93, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L1 (could bias magnitude, not mitigated): The accuracy measure of the M4 hourly tests changed after their results were seen: MASE decided the tests of C4, C5, C6 and C9 (lab run RN8), the guarantor then judged MASE unsuitable and set RMSSE (decision DC4 of the lab record), and C10 to C13 restate those hypotheses on RMSSE and test them on the same M4 test split (run RN11), so they are not independent of C4 to C6 and C9. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C6: 93, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L1 (could bias magnitude, not mitigated): The accuracy measure of the M4 hourly tests changed after their results were seen: MASE decided the tests of C4, C5, C6 and C9 (lab run RN8), the guarantor then judged MASE unsuitable and set RMSSE (decision DC4 of the lab record), and C10 to C13 restate those hypotheses on RMSSE and test them on the same M4 test split (run RN11), so they are not independent of C4 to C6 and C9. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C7: 93, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L4 (could bias magnitude, not mitigated): Elapsed times depend on the machine and on the language: forecast::tbats runs in R and the other methods in Python, the M4 runs ran on a cloud machine and the electricity runs on the guarantor's computer, and each time includes its language's overhead per fit. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C8: 100, Well supported

Novelty: new, by the authors

Nothing lost.

C9: 94, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L1 (could bias magnitude, not mitigated): The accuracy measure of the M4 hourly tests changed after their results were seen: MASE decided the tests of C4, C5, C6 and C9 (lab run RN8), the guarantor then judged MASE unsuitable and set RMSSE (decision DC4 of the lab record), and C10 to C13 restate those hypotheses on RMSSE and test them on the same M4 test split (run RN11), so they are not independent of C4 to C6 and C9. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C10: 89, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L1 (could bias magnitude, not mitigated): The accuracy measure of the M4 hourly tests changed after their results were seen: MASE decided the tests of C4, C5, C6 and C9 (lab run RN8), the guarantor then judged MASE unsuitable and set RMSSE (decision DC4 of the lab record), and C10 to C13 restate those hypotheses on RMSSE and test them on the same M4 test split (run RN11), so they are not independent of C4 to C6 and C9. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −3 Plan The lab record flags the test: code changed after plan. To remove it: Rerun the test as planned, or explain the flag in a limitation.
  3. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  4. −1 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C11: 92, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L1 (could bias magnitude, not mitigated): The accuracy measure of the M4 hourly tests changed after their results were seen: MASE decided the tests of C4, C5, C6 and C9 (lab run RN8), the guarantor then judged MASE unsuitable and set RMSSE (decision DC4 of the lab record), and C10 to C13 restate those hypotheses on RMSSE and test them on the same M4 test split (run RN11), so they are not independent of C4 to C6 and C9. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −1 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C12: 92, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L1 (could bias magnitude, not mitigated): The accuracy measure of the M4 hourly tests changed after their results were seen: MASE decided the tests of C4, C5, C6 and C9 (lab run RN8), the guarantor then judged MASE unsuitable and set RMSSE (decision DC4 of the lab record), and C10 to C13 restate those hypotheses on RMSSE and test them on the same M4 test split (run RN11), so they are not independent of C4 to C6 and C9. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −1 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C13: 90, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L1 (could bias magnitude, not mitigated): The accuracy measure of the M4 hourly tests changed after their results were seen: MASE decided the tests of C4, C5, C6 and C9 (lab run RN8), the guarantor then judged MASE unsuitable and set RMSSE (decision DC4 of the lab record), and C10 to C13 restate those hypotheses on RMSSE and test them on the same M4 test split (run RN11), so they are not independent of C4 to C6 and C9. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −3 Plan The lab record flags the test: code changed after plan. To remove it: Rerun the test as planned, or explain the flag in a limitation.
  3. −1 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C14: 92, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L3 (could bias magnitude, not mitigated): The electricity evaluation covers one week, the 7 daily origins before 2014-09-08, of the clients of one utility; other weeks or seasons of the year may rank the methods differently. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −1 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C15: 92, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L3 (could bias magnitude, not mitigated): The electricity evaluation covers one week, the 7 daily origins before 2014-09-08, of the clients of one utility; other weeks or seasons of the year may rank the methods differently. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −1 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C16: 92, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L3 (could bias magnitude, not mitigated): The electricity evaluation covers one week, the 7 daily origins before 2014-09-08, of the clients of one utility; other weeks or seasons of the year may rank the methods differently. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −1 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C17: 90, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L3 (could bias magnitude, not mitigated): The electricity evaluation covers one week, the 7 daily origins before 2014-09-08, of the clients of one utility; other weeks or seasons of the year may rank the methods differently. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −3 Plan The lab record flags the test: code changed after plan. To remove it: Rerun the test as planned, or explain the flag in a limitation.
  3. −1 Open problems Limitation L5 (reduces precision, not mitigated): The lab record flags plans PL1, PL5 and PL7 with code changed after the plan; between PL1 and its run only m4_hourly.py changed, not the script the run used; between PL5 and its run only electricity_hourly.py was added; between PL7 and its run the change set the null hypothesis of superiority that PL7's design specifies. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C18: 93, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L3 (could bias magnitude, not mitigated): The electricity evaluation covers one week, the 7 daily origins before 2014-09-08, of the clients of one utility; other weeks or seasons of the year may rank the methods differently. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C19: 93, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L3 (could bias magnitude, not mitigated): The electricity evaluation covers one week, the 7 daily origins before 2014-09-08, of the clients of one utility; other weeks or seasons of the year may rank the methods differently. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C20: 90, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L3 (could bias magnitude, not mitigated): The electricity evaluation covers one week, the 7 daily origins before 2014-09-08, of the clients of one utility; other weeks or seasons of the year may rank the methods differently. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −3 Open problems Limitation L4 (could bias magnitude, not mitigated): Elapsed times depend on the machine and on the language: forecast::tbats runs in R and the other methods in Python, the M4 runs ran on a cloud machine and the electricity runs on the guarantor's computer, and each time includes its language's overhead per fit. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −1 Open problems Limitation L2 (reduces precision, not mitigated): The t-tests and their intervals treat the series as independent; the series of one data set share calendar and weather effects, so the intervals may be too narrow. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C21: 100, Well supported

Novelty: new, by the authors

Nothing lost.

C23: 100, Well supported

Novelty: new, by the authors

Nothing lost.

C24: 100, Well supported

Novelty: new, by the authors

Nothing lost.

Versions

  • Version 2 (this page) · 13 November 2026 at 12:34 UTC · updated
  • Version 1 · 13 November 2026 at 12:04 UTC · active

Every version is kept in the public registry (github.com/entrylib/registry).