{ "bundle_of": "C1", "schema_version": "0.42", "paper": { "id": "entrylib:2026.000007", "version": 2, "title": "TBATS in smooth: its formulation, and its accuracy against forecast::tbats and AutoTBATS on hourly data", "licence": "CC-BY-4.0", "status": "corrected", "date": "2026-11-13", "status_notice": { "date": "2026-11-13", "reason_code": "correction", "statement": "The paper had no summary and did not state why the ARMA part of smooth's TBATS differs from De Livera's in effect: a summary in plain language is added; a formal claim (C21, with its condition C22 and proof E6) shows that ARMA errors pass the ARMA's persistence into the level, while ADAM's additive ARMA component leaves the level a fixed share of each shock; the highlights are amended; and the equation of the harmonic states in D1 is set on a line of its own.", "issued_by": "Ivan Svetunkov (ORCID 0000-0001-7826-0281)" } }, "claims": [ { "id": "C1", "statement": "[[D1:smooth TBATS]] without ARMA is the model of [[D2:De Livera TBATS]]: with the same smoothing parameters, damping, Box-Cox parameter and initial states, its one-step fitted values in the Box-Cox space differ from those of an independent implementation of De Livera's recursion by at most 5.1e-14 in relative terms over the 360 cases of the [[D14:formulation checks]].", "about": { "subject": "D1", "relation": "gives the same fitted values as", "object": "D2", "comparator": null, "conditions": [ "same parameters and initial states, no ARMA" ] }, "type": "methodological", "role": "principal", "novelty": "no_prior_found", "support": [ { "kind": "simulation", "evidence": "E1", "results": [ "E1.R1" ], "artefacts": [ "A10" ] } ], "provenance": "author_stated", "hypothesis": { "lab_event": "H1", "outcome": "supported" }, "planning": "confirmatory", "planned_at": "2026-11-11T13:16:58Z", "planned_at_anchor": "git_remote", "test": { "framework": "deterministic", "parameters": { "tolerance": 1e-08, "refutation": 1e-06, "machine_tolerance": 1e-12, "sample": "10 seeds (101-110), one run, no stopping rule; the checks are deterministic given the seeds" }, "quantities": { "h1": { "variable": "max relative difference of fitted values over all cases, smooth TBATS vs De Livera recursion" } }, "supported": "h1 < tolerance", "not_supported": "h1 >= refutation", "verdict": "computed", "runs": 1, "lab_plan": "PL1", "flags": [ "code_changed_after_plan" ] }, "scope": { "data_type": "simulated series", "conditions": [ "seeds 101 to 110", "series of 300 observations", "periods 24, 7.3, 168 and 30.4", "Box-Cox parameter 1, 0.5 and 0", "no, additive and damped trend" ] }, "relations": [ { "type": "extends", "reference": "B4", "note": "De Livera, Hyndman and Snyder introduced TBATS; this claim is about the implementation of smooth, which places the harmonics in ADAM's state space core, compared with two other implementations." } ], "assessments": [ { "by": "author", "strength": "high", "rationale": "360 cases over four sets of periods, three trends and three Box-Cox parameters, all within 1e-13 of an independent implementation." } ], "verifiability": "open", "verifiability_flags": [] } ], "definitions": [ { "id": "D1", "term": "smooth TBATS", "kind": "method", "definition": "TBATS of the smooth package (R `tbats()`, Python `TBATS`): a linear single-source-of-error state space model of the Box-Cox transform $y_t^{(\\lambda)}$ of the series, built on ADAM's state space core,\n$$\\begin{aligned} y_t^{(\\lambda)} &= w^\\top v_{t-\\boldsymbol{l}} + \\epsilon_t, \\\\ v_t &= F v_{t-\\boldsymbol{l}} + g \\epsilon_t, \\end{aligned}$$\nwhere $v_{t-\\boldsymbol{l}}$ takes each state at its own lag. The level $l_t$ and the trend $b_t$ are those of ETS(A,N,N), ETS(A,A,N) or ETS(A,Ad,N),\n$$\\begin{aligned} l_t &= l_{t-1} + \\phi b_{t-1} + \\alpha \\epsilon_t, \\\\ b_t &= \\phi b_{t-1} + \\beta \\epsilon_t. \\end{aligned}$$\nEach harmonic $j = 1, \\dots, k_i$ of each period $m_i$, with frequency $\\lambda_{ij} = 2 \\pi j / m_i$, is two states $v^{(ij)}_1$ and $v^{(ij)}_2$ with lags 1 and 2 in the ARIMA slot of ADAM's core,\n$$\\begin{aligned} u_{ij,t} &= v^{(ij)}_{1,t-1} + v^{(ij)}_{2,t-2}, \\\\ v^{(ij)}_{1,t} &= 2 \\cos(\\lambda_{ij}) \\, u_{ij,t} + \\gamma_{1i} \\epsilon_t, \\\\ v^{(ij)}_{2,t} &= -u_{ij,t} + \\left(\\sin(\\lambda_{ij}) \\gamma_{2i} - \\cos(\\lambda_{ij}) \\gamma_{1i}\\right) \\epsilon_t, \\end{aligned}$$\nso that, with the backshift operator $B$,\n$$(1 - 2 \\cos(\\lambda_{ij}) B + B^2) \\, u_{ij,t} = \\left(\\gamma_{1i} B + (\\sin(\\lambda_{ij}) \\gamma_{2i} - \\cos(\\lambda_{ij}) \\gamma_{1i}) B^2\\right) \\epsilon_t.$$\nThe measurement equation is\n$$y_t^{(\\lambda)} = l_{t-1} + \\phi b_{t-1} + \\sum_i \\sum_{j=1}^{k_i} u_{ij,t} + a_t + \\epsilon_t,$$\nwhere $a_t$ is an optional ARMA component of ADAM: an additive block of states whose output enters the measurement equation, not ARMA errors. $\\lambda \\in [0, 1]$ is estimated by likelihood with the Jacobian of the transform. The harmonics are preselected by information criterion on a global regression of the transformed series on Fourier terms, and the trend and the ARMA orders are selected by AICc. With the default `distribution=\"auto\"`, the structure is selected with the generalised normal distribution of $\\epsilon_t$, and the closest of the normal, Laplace and S distributions to its estimated shape is then fitted on that structure. Code: R/adam-tbats.R and python/src/smooth/adam_general/core/tbats/ of the smooth repository, at the code versions the evidence names." }, { "id": "D2", "term": "De Livera TBATS", "kind": "method", "definition": "TBATS of De Livera, Hyndman and Snyder (2011): for the Box-Cox transform $y_t^{(\\lambda)}$,\n$$\\begin{aligned} y_t^{(\\lambda)} &= l_{t-1} + \\phi b_{t-1} + \\sum_i \\sum_{j=1}^{k_i} s^{(i)}_{j,t-1} + d_t, \\\\ l_t &= l_{t-1} + \\phi b_{t-1} + \\alpha d_t, \\\\ b_t &= \\phi b_{t-1} + \\beta d_t, \\\\ s^{(i)}_{j,t} &= \\cos(\\lambda_{ij}) s^{(i)}_{j,t-1} + \\sin(\\lambda_{ij}) s^{*(i)}_{j,t-1} + \\gamma_{1i} d_t, \\\\ s^{*(i)}_{j,t} &= -\\sin(\\lambda_{ij}) s^{(i)}_{j,t-1} + \\cos(\\lambda_{ij}) s^{*(i)}_{j,t-1} + \\gamma_{2i} d_t, \\\\ d_t &= \\sum_{k=1}^{p} \\varphi_k d_{t-k} + \\sum_{k=1}^{q} \\theta_k \\epsilon_{t-k} + \\epsilon_t, \\end{aligned}$$\nwith $\\lambda_{ij} = 2 \\pi j / m_i$: ARMA(p, q) errors $d_t$ drive every state.", "source": "B4" }, { "id": "D3", "term": "forecast::tbats", "kind": "method", "definition": "The tbats() function of the R package forecast (Hyndman et al.), De Livera et al.'s TBATS with its own model selection, called with seasonal.periods = c(24, 168), single-threaded, otherwise default settings.", "source": "B5" }, { "id": "D4", "term": "AutoTBATS", "kind": "method", "definition": "AutoTBATS of the Python package statsforecast (Nixtla), called with season_length = [24, 168], otherwise default settings.", "source": "B12" }, { "id": "D5", "term": "ADAM ETS", "kind": "method", "definition": "ADAM ETS of smooth's Python port, ADAM(model='ZXZ', lags=[24, 168]) with its default settings: ETS with a daily and a weekly seasonal component selected by AICc.", "source": "B8" }, { "id": "D6", "term": "seasonal naive 168", "kind": "method", "definition": "Seasonal naive forecast repeating the last 168 observations (the last week) of the training sample." }, { "id": "D7", "term": "M4 hourly", "kind": "data", "definition": "The 414 hourly series of the M4 competition (Makridakis et al., 2020), training samples of 700 or 960 observations and test samples of 48, from the Mcompetitions/M4-methods repository (Dataset/Train/Hourly-train.csv, Dataset/Test/Hourly-test.csv).", "source": "B6" }, { "id": "D9", "term": "fixed-origin evaluation", "kind": "design", "definition": "Each method is fitted once to each training sample and forecasts h = 1..48 steps ahead from its end; no refits, no rolling origin." }, { "id": "D10", "term": "MASE", "kind": "variable", "definition": "Mean absolute scaled error of a series over the horizon $h = 48$,\n$$\\mathrm{MASE} = \\frac{\\frac{1}{h} \\sum_{j=1}^{h} |y_{n+j} - \\hat y_{n+j}|}{\\frac{1}{n-24} \\sum_{t=25}^{n} |y_t - y_{t-24}|},$$\nscaled by the in-sample mean absolute seasonal difference at lag 24, as in M4.", "source": "B10" }, { "id": "D11", "term": "relative MASE", "kind": "variable", "definition": "Geometric mean over series of the ratio of the [[D10:MASE]] of method A to that of method B; below 1, A is more accurate. Its 99% confidence interval is the exponential of the t-interval of the mean log ratio over series." }, { "id": "D12", "term": "coverage", "kind": "variable", "definition": "Share of the test values of a set of series and horizons that lie inside the 95% prediction intervals of a method." }, { "id": "D13", "term": "elapsed time", "kind": "variable", "definition": "Wall-clock seconds to fit a method to one training sample and produce its forecasts and intervals, single-threaded." }, { "id": "D14", "term": "formulation checks", "kind": "design", "definition": "Deterministic checks of smooth TBATS against its specification, per seed: (H1) random parameters, initial states and series of 300 observations simulated by De Livera's recursion with normal errors, for the periods {24}, {7.3}, {24, 168}, {7.3, 30.4}, no/additive/damped trend and lambda in {1, 0.5, 0}, fitted by smooth TBATS with all parameters and initial states fixed; (H2) a noise-free series of 300 observations made of a random level and random harmonics of the periods 7.3 and 24, fitted with random fixed persistence and initial='backcasting'; (H3) nine random pairs of alpha in (0.05, 0.95) and |phi| in (0.05, 0.95) for a local level with AR(1), the impulse response w'F^(j-1)g at j = 1..20 read from the fitted model's matrices. Code: claimset-code/smooth-tbats/formulation_checks.py." }, { "id": "D15", "term": "Nemenyi test", "kind": "design", "definition": "The Nemenyi test as computed by greybox's rmcb (distribution 'tukey', ties averaged): the methods are ranked within each series by the error measure, and two methods differ at level $1 - \\alpha$ when their mean ranks over series differ by more than the critical distance\n$$\\mathrm{CD} = \\frac{q_{1-\\alpha}(k, \\infty)}{\\sqrt{2}} \\sqrt{\\frac{k (k + 1)}{6 N}}$$\nfor $k$ methods and $N$ series.", "source": "B11" }, { "id": "D16", "term": "RMSSE", "kind": "variable", "definition": "Root mean squared scaled error of a series over the horizon $h = 48$,\n$$\\mathrm{RMSSE} = \\sqrt{\\frac{\\frac{1}{h} \\sum_{j=1}^{h} (y_{n+j} - \\hat y_{n+j})^2}{\\frac{1}{n-1} \\sum_{t=2}^{n} (y_t - y_{t-1})^2}},$$\nscaled by the in-sample root mean squared first difference of the fitting sample, as in M5.", "source": "B7" }, { "id": "D17", "term": "relative RMSSE", "kind": "variable", "definition": "Geometric mean over series of the ratio of the [[D16:RMSSE]] of method A to that of method B; below 1, A is more accurate. Its 99% confidence interval is the exponential of the t-interval of the mean log ratio over series." }, { "id": "D18", "term": "electricity hourly", "kind": "data", "definition": "The 370 hourly series of electricity consumption (kW) of clients of a Portuguese utility in GluonTS's electricity_nips archive (SR3), drawn from UCI ElectricityLoadDiagrams20112014: each series is the archive's training series extended by its last test window, ending 2014-09-08 01:00, 1249 to 6001 hours long.", "source": "B9" }, { "id": "D19", "term": "rolling-origin evaluation", "kind": "design", "definition": "Seven daily forecast origins over the last week of each series: the last one 48 hours before its end, the others 24, 48, ..., 144 hours earlier. At each origin each method is fitted to the 1008 preceding hours (six weeks) and forecasts 48 hours ahead; the RMSSE and the interval coverage of a series are averaged over its seven origins." }, { "id": "D20", "term": "smooth TBATS dnorm", "kind": "method", "definition": "[[D1:smooth TBATS]] with the normal distribution of $\\epsilon_t$: `TBATS(lags=[1, 24, 168], distribution=\"dnorm\")` of the Python smooth package, every other setting at its default. The structure is selected and fitted with the normal distribution only, instead of the generalised normal distribution followed by the closest named distribution of the default `distribution=\"auto\"`.", "broader": [ "D1" ] } ], "evidence": [ { "id": "E1", "profile": "simulation", "description": "The [[D14:formulation checks]] of [[D1:smooth TBATS]], run once with the seeds 101 to 110 (lab run RN6, plan PL1): the largest relative difference of the one-step fitted values from an independent implementation of the recursion of [[D2:De Livera TBATS]] over 360 cases (H1), the largest absolute in-sample error of backcasting on 10 noise-free series (H2), and, over nine pairs of smoothing parameter and AR(1) coefficient, the largest distance of the impulse response of smooth TBATS from that of an additive ARMA component and the smallest from that of ARMA errors (H3). The per-case values are in the CSV of the lab results (A8).", "results": [ { "id": "R1", "variable": "max relative difference of fitted values over all cases, smooth TBATS vs De Livera recursion", "value": 5.1122857914327425e-14, "conditions": { "seeds": "[101, 102, 103, 104, 105, 106, 107, 108, 109, 110]", "cases": 360 }, "statistic": "max" }, { "id": "R2", "variable": "max absolute in-sample error over all noise-free series, backcasting", "value": 1.7696635268293903e-09, "conditions": { "seeds": "[101, 102, 103, 104, 105, 106, 107, 108, 109, 110]", "n": 300 }, "statistic": "max" }, { "id": "R3", "variable": "max absolute difference of smooth's impulse response from alpha + phi^j over all cases", "value": 2.220446049250313e-16, "statistic": "max" }, { "id": "R4", "variable": "min over cases of the max absolute difference of smooth's impulse response from the ARMA-errors response", "value": 0.01651176558114527, "statistic": "min" } ], "artefacts": [ "A8", "A10" ], "lab_runs": [ "RN6" ], "setup": { "studied": [ "D1", "D2" ], "procedure": [ "D14" ] }, "execution": { "code_version": "3444ddada1be34139353afffdc25569fafd4f71a", "software": { "python": "3.11.18", "platform": "Linux-6.18.44-fc-v378-x86_64-with-glibc2.39" } }, "availability": { "level": "public" } }, { "id": "E2", "profile": "quant", "description": "Benchmark of [[D1:smooth TBATS]] against [[D3:forecast::tbats]], [[D4:AutoTBATS]], [[D5:ADAM ETS]] and [[D6:seasonal naive 168]] on the 414 series of [[D7:M4 hourly]] with [[D9:fixed-origin evaluation]] (the M4 test split, 48 hours ahead), run once under plan PL3 (lab run RN8). Each fit is single-threaded; a failed fit gets the seasonal naive 168 forecasts and no interval. Per series, [[D10:MASE]], the coverage of the 95% interval and the elapsed time (averaged over the origins where there are several); the relative MASE of smooth TBATS to each comparator is the geometric mean over series of the ratio of their MASE, with a 99% t-interval on the log scale; the one-sided t-tests on the mean log ratios are adjusted by Holm over four; the [[D15:Nemenyi test]] ranks the five methods within each series. The sMAPE that the run also computed is in the CSV of the lab results only (decision DC4 of the lab record).", "results": [ { "id": "R1", "variable": "D11", "value": 0.8317576290033031, "interval": { "level": 0.99, "lower": 0.7793039397579915, "upper": 0.8877418913345124, "kind": "confidence" }, "conditions": { "comparator": "D3" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R2", "variable": "D11", "value": 0.780315891451513, "interval": { "level": 0.99, "lower": 0.7242363686872306, "upper": 0.840737798842475, "kind": "confidence" }, "conditions": { "comparator": "D4" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R3", "variable": "D11", "value": 0.9260655272734568, "interval": { "level": 0.99, "lower": 0.824281045050103, "upper": 1.0404186362821646, "kind": "confidence" }, "conditions": { "comparator": "D5" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R4", "variable": "D11", "value": 0.5067448313602549, "interval": { "level": 0.99, "lower": 0.44786668083806935, "upper": 0.5733633134525995, "kind": "confidence" }, "conditions": { "comparator": "D6" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R5", "variable": "relative elapsed time", "value": 0.5790894532352115, "interval": { "level": 0.99, "lower": 0.5389759363891272, "upper": 0.6221884358973419, "kind": "confidence" }, "conditions": { "comparator": "D3" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R6", "variable": "critical distance, Nemenyi test (MASE)", "value": 0.35767939611022115, "conditions": { "level": 0.99 } }, { "id": "R7", "variable": "series in Nemenyi test", "value": 414, "statistic": "count" }, { "id": "R8", "variable": "p-value, rmcb test of equal mean ranks (MASE)", "value": 1.1039717573166142e-32, "statistic": "p_value" } ], "tables": [ { "id": "T1", "description": "Per method: mean and median MASE over series, coverage of the 95% prediction intervals, median elapsed time per series in seconds, series with a failed fit, and the mean rank of MASE in the Nemenyi test with the bounds of its 99% interval.", "row_labels": [ "method" ], "columns": [ { "variable": "D10", "statistic": "mean" }, { "variable": "D10", "statistic": "median" }, { "variable": "D12", "conditions": { "nominal level": 0.95 }, "statistic": "mean" }, { "variable": "D13", "statistic": "median" }, { "variable": "series failed", "statistic": "count" }, { "variable": "mean rank of MASE, Nemenyi test", "statistic": "mean" }, { "variable": "mean rank of MASE, Nemenyi test", "conditions": { "bound": "99% interval, lower" } }, { "variable": "mean rank of MASE, Nemenyi test", "conditions": { "bound": "99% interval, upper" } } ], "rows": [ { "id": "r1", "label": [ "D1" ], "values": [ 1.029, 0.87, 0.9273349436392914, 14.18, 0, 2.5724637681159437, 2.394, 2.751 ] }, { "id": "r2", "label": [ "D3" ], "values": [ 1.304, 0.9941, 0.9101, 24.4, 0, 2.9541062801932383, 2.775, 3.133 ] }, { "id": "r3", "label": [ "D4" ], "values": [ 1.516, 0.9836, 0.9183, 21.42, 0, 3.07729468599034, 2.898, 3.256 ] }, { "id": "r4", "label": [ "D5" ], "values": [ 2.11, 0.7984, 0.9054, 1.31, 0, 2.618, 2.44, 2.797 ] }, { "id": "r5", "label": [ "D6" ], "values": [ 2.867, 1.397, null, 4.323e-05, 0, 3.778, 3.599, 3.957 ] } ], "artefact": "A8" }, { "id": "T2", "description": "Per comparator: the series the relative MASE of smooth TBATS to it is computed on, and those left out because the MASE of smooth TBATS or of the comparator is zero.", "conditions": { "method": "D1" }, "row_labels": [ "comparator" ], "columns": [ { "variable": "series in relative MASE", "statistic": "count" }, { "variable": "series excluded by zero MASE", "statistic": "count" } ], "rows": [ { "id": "r1", "label": [ "D3" ], "values": [ 414, 0 ] }, { "id": "r2", "label": [ "D4" ], "values": [ 414, 0 ] }, { "id": "r3", "label": [ "D5" ], "values": [ 414, 0 ] }, { "id": "r4", "label": [ "D6" ], "values": [ 414, 0 ] } ], "artefact": "A8" }, { "id": "T3", "description": "The one-sided t-tests of the plan on the per-series log ratios of smooth TBATS to the comparator: p-value and p-value adjusted by Holm over the four tests.", "row_labels": [ "comparator", "measure" ], "columns": [ { "variable": "p-value, one-sided t-test", "statistic": "p_value" }, { "variable": "Holm-adjusted p-value, one-sided t-test", "statistic": "p_value" } ], "rows": [ { "id": "r1", "label": [ "D3", "MASE" ], "values": [ 5.799e-19, 5.798511501263025e-19 ] }, { "id": "r2", "label": [ "D4", "MASE" ], "values": [ 1.349e-22, 2.6978102344588355e-22 ] }, { "id": "r3", "label": [ "D6", "MASE" ], "values": [ 5.031e-38, 1.5092330045971394e-37 ] }, { "id": "r4", "label": [ "D3", "elapsed time" ], "values": [ 1.154e-61, 4.6151072509330304e-61 ] } ], "artefact": "A8" } ], "artefacts": [ "A8", "A10" ], "lab_runs": [ "RN8" ], "setup": { "studied": [ "D1", "D3", "D4", "D5", "D6" ], "cases": [ "D7" ], "procedure": [ "D9", "D15" ], "lens": [ "D10", "D11", "D12", "D13" ] }, "execution": { "code_version": "3444ddada1be34139353afffdc25569fafd4f71a", "software": { "python": "3.11.18", "platform": "Linux-6.18.44-fc-v378-x86_64-with-glibc2.39" } }, "availability": { "level": "public" }, "data_origin": "secondary" }, { "id": "E3", "profile": "quant", "description": "Benchmark of [[D1:smooth TBATS]] against [[D3:forecast::tbats]], [[D4:AutoTBATS]], [[D5:ADAM ETS]] and [[D6:seasonal naive 168]] on the 414 series of [[D7:M4 hourly]] with [[D9:fixed-origin evaluation]] (the M4 test split, 48 hours ahead), run once under plan PL5 (lab run RN11). Each fit is single-threaded; a failed fit gets the seasonal naive 168 forecasts and no interval. Per series, [[D16:RMSSE]], the coverage of the 95% interval and the elapsed time (averaged over the origins where there are several); the relative RMSSE of smooth TBATS to each comparator is the geometric mean over series of the ratio of their RMSSE, with a 99% t-interval on the log scale; the one-sided t-tests on the mean log ratios are adjusted by Holm over four; the [[D15:Nemenyi test]] ranks the five methods within each series.", "results": [ { "id": "R1", "variable": "D17", "value": 0.8480372827132892, "interval": { "level": 0.99, "lower": 0.7969221555458947, "upper": 0.9024309687802653, "kind": "confidence" }, "conditions": { "comparator": "D3" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R2", "variable": "D17", "value": 0.7959442609119876, "interval": { "level": 0.99, "lower": 0.741071448751255, "upper": 0.854880143535765, "kind": "confidence" }, "conditions": { "comparator": "D4" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R3", "variable": "D17", "value": 0.9482580389950931, "interval": { "level": 0.99, "lower": 0.8470034046237921, "upper": 1.0616171122927285, "kind": "confidence" }, "conditions": { "comparator": "D5" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R4", "variable": "D17", "value": 0.535834976024436, "interval": { "level": 0.99, "lower": 0.47846285660893406, "upper": 0.600086542905422, "kind": "confidence" }, "conditions": { "comparator": "D6" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R5", "variable": "relative elapsed time", "value": 0.568175851157546, "interval": { "level": 0.99, "lower": 0.5281131840262834, "upper": 0.6112776722925655, "kind": "confidence" }, "conditions": { "comparator": "D3" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R6", "variable": "critical distance, Nemenyi test (RMSSE)", "value": 0.35767939611022115, "conditions": { "level": 0.99 } }, { "id": "R7", "variable": "series in Nemenyi test", "value": 414, "statistic": "count" }, { "id": "R8", "variable": "p-value, rmcb test of equal mean ranks (RMSSE)", "value": 4.096753399289539e-34, "statistic": "p_value" } ], "tables": [ { "id": "T1", "description": "Per method: mean and median RMSSE over series, coverage of the 95% prediction intervals, median elapsed time per series in seconds, series with a failed fit, and the mean rank of RMSSE in the Nemenyi test with the bounds of its 99% interval.", "row_labels": [ "method" ], "columns": [ { "variable": "D16", "statistic": "mean" }, { "variable": "D16", "statistic": "median" }, { "variable": "D12", "conditions": { "nominal level": 0.95 }, "statistic": "mean" }, { "variable": "D13", "statistic": "median" }, { "variable": "series failed", "statistic": "count" }, { "variable": "mean rank of RMSSE, Nemenyi test", "statistic": "mean" }, { "variable": "mean rank of RMSSE, Nemenyi test", "conditions": { "bound": "99% interval, lower" } }, { "variable": "mean rank of RMSSE, Nemenyi test", "conditions": { "bound": "99% interval, upper" } } ], "rows": [ { "id": "r1", "label": [ "D1" ], "values": [ 0.8294, 0.6242, 0.9273, 12.25, 0, 2.562801932367151, 2.384, 2.742 ] }, { "id": "r2", "label": [ "D3" ], "values": [ 0.9094, 0.698, 0.9101, 21.84, 0, 2.9371980676328517, 2.758, 3.116 ] }, { "id": "r3", "label": [ "D4" ], "values": [ 1.063, 0.6854, 0.9183, 19.7, 0, 3.0434782608695667, 2.865, 3.222 ] }, { "id": "r4", "label": [ "D5" ], "values": [ 2.1, 0.6314, 0.9055, 1.112, 0, 2.645, 2.466, 2.824 ] }, { "id": "r5", "label": [ "D6" ], "values": [ 1.453, 1.095, null, 4.024e-05, 0, 3.812, 3.633, 3.99 ] } ], "artefact": "A8" }, { "id": "T2", "description": "Per comparator: the series the relative RMSSE of smooth TBATS to it is computed on, and those left out because the RMSSE of smooth TBATS or of the comparator is zero.", "conditions": { "method": "D1" }, "row_labels": [ "comparator" ], "columns": [ { "variable": "series in relative RMSSE", "statistic": "count" }, { "variable": "series excluded by zero RMSSE", "statistic": "count" } ], "rows": [ { "id": "r1", "label": [ "D3" ], "values": [ 414, 0 ] }, { "id": "r2", "label": [ "D4" ], "values": [ 414, 0 ] }, { "id": "r3", "label": [ "D5" ], "values": [ 414, 0 ] }, { "id": "r4", "label": [ "D6" ], "values": [ 414, 0 ] } ], "artefact": "A8" }, { "id": "T3", "description": "The one-sided t-tests of the plan on the per-series log ratios of smooth TBATS to the comparator: p-value and p-value adjusted by Holm over the four tests.", "row_labels": [ "comparator", "measure" ], "columns": [ { "variable": "p-value, one-sided t-test", "statistic": "p_value" }, { "variable": "Holm-adjusted p-value, one-sided t-test", "statistic": "p_value" } ], "rows": [ { "id": "r1", "label": [ "D3", "RMSSE" ], "values": [ 9.427e-18, 9.427033177562526e-18 ] }, { "id": "r2", "label": [ "D4", "RMSSE" ], "values": [ 1.181e-21, 2.3628603624600894e-21 ] }, { "id": "r3", "label": [ "D6", "RMSSE" ], "values": [ 4.342e-38, 1.302649972219819e-37 ] }, { "id": "r4", "label": [ "D3", "elapsed time" ], "values": [ 4.715e-63, 1.886e-62 ] } ], "artefact": "A8" } ], "artefacts": [ "A8", "A10" ], "lab_runs": [ "RN11" ], "setup": { "studied": [ "D1", "D3", "D4", "D5", "D6" ], "cases": [ "D7" ], "procedure": [ "D9", "D15" ], "lens": [ "D16", "D17", "D12", "D13" ] }, "execution": { "code_version": "1f24c10d7a472c654346afcc2c7158f9982e0659", "software": { "python": "3.11.18", "platform": "Linux-6.18.44-fc-v378-x86_64-with-glibc2.39" } }, "availability": { "level": "public" }, "data_origin": "secondary" }, { "id": "E4", "profile": "quant", "description": "Benchmark of [[D1:smooth TBATS]] against [[D3:forecast::tbats]], [[D4:AutoTBATS]], [[D5:ADAM ETS]] and [[D6:seasonal naive 168]] on the 370 series of [[D18:electricity hourly]] with [[D19:rolling-origin evaluation]] (7 daily origins of the last week, 1008-hour fitting windows, 48 hours ahead; series 322, constant in its windows, has an undefined RMSSE and is left out), run once under plan PL7 (lab run RN13). Each fit is single-threaded; a failed fit gets the seasonal naive 168 forecasts and no interval. Per series, [[D16:RMSSE]], the coverage of the 95% interval and the elapsed time (averaged over the origins where there are several); the relative RMSSE of smooth TBATS to each comparator is the geometric mean over series of the ratio of their RMSSE, with a 99% t-interval on the log scale; the one-sided t-tests on the mean log ratios are adjusted by Holm over four; the [[D15:Nemenyi test]] ranks the five methods within each series.", "results": [ { "id": "R1", "variable": "D17", "value": 0.9752899990850776, "interval": { "level": 0.99, "lower": 0.9470396739420402, "upper": 1.0043830353548469, "kind": "confidence" }, "conditions": { "comparator": "D3" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R2", "variable": "D17", "value": 0.990872986502657, "interval": { "level": 0.99, "lower": 0.9753390861558684, "upper": 1.006654290099668, "kind": "confidence" }, "conditions": { "comparator": "D4" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R3", "variable": "D17", "value": 1.0042706046509193, "interval": { "level": 0.99, "lower": 0.7972644036763257, "upper": 1.2650250565750576, "kind": "confidence" }, "conditions": { "comparator": "D5" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R4", "variable": "D17", "value": 0.8243857583084898, "interval": { "level": 0.99, "lower": 0.7980923108751354, "upper": 0.8515454531276542, "kind": "confidence" }, "conditions": { "comparator": "D6" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R5", "variable": "relative elapsed time", "value": 0.6094493108259951, "interval": { "level": 0.99, "lower": 0.5799927267313179, "upper": 0.6404019315199876, "kind": "confidence" }, "conditions": { "comparator": "D3" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R6", "variable": "critical distance, Nemenyi test (RMSSE)", "value": 0.37886188095533324, "conditions": { "level": 0.99 } }, { "id": "R7", "variable": "series in Nemenyi test", "value": 369, "statistic": "count" }, { "id": "R8", "variable": "p-value, rmcb test of equal mean ranks (RMSSE)", "value": 1.5420804210195766e-81, "statistic": "p_value" }, { "id": "R9", "variable": "series excluded by undefined RMSSE", "value": 1, "statistic": "count" } ], "tables": [ { "id": "T1", "description": "Per method: mean and median RMSSE over series, coverage of the 95% prediction intervals, median elapsed time per series in seconds, series with a failed fit, and the mean rank of RMSSE in the Nemenyi test with the bounds of its 99% interval.", "row_labels": [ "method" ], "columns": [ { "variable": "D16", "statistic": "mean" }, { "variable": "D16", "statistic": "median" }, { "variable": "D12", "conditions": { "nominal level": 0.95 }, "statistic": "mean" }, { "variable": "D13", "statistic": "median" }, { "variable": "series failed", "statistic": "count" }, { "variable": "fits failed", "statistic": "count" }, { "variable": "mean rank of RMSSE, Nemenyi test", "statistic": "mean" }, { "variable": "mean rank of RMSSE, Nemenyi test", "conditions": { "bound": "99% interval, lower" } }, { "variable": "mean rank of RMSSE, Nemenyi test", "conditions": { "bound": "99% interval, upper" } } ], "rows": [ { "id": "r1", "label": [ "D1" ], "values": [ 0.7948, 0.6644, 0.954, 12.22, 0, 0, 2.4823848238482347, 2.293, 2.672 ] }, { "id": "r2", "label": [ "D3" ], "values": [ 0.7985, 0.6878, 0.9433, 20.26, 0, 0, 2.558265582655823, 2.369, 2.748 ] }, { "id": "r3", "label": [ "D4" ], "values": [ 0.7949, 0.671, 0.9372, 15.07, 0, 0, 2.38753387533875, 2.198, 2.577 ] }, { "id": "r4", "label": [ "D5" ], "values": [ 0.8795, 0.7065, 0.9776, 1.245, 0, 0, 3.276, 3.087, 3.466 ] }, { "id": "r5", "label": [ "D6" ], "values": [ 1.003, 0.8002, null, 3.08e-05, 0, 0, 4.295, 4.106, 4.485 ] } ], "artefact": "A8" }, { "id": "T2", "description": "Per comparator: the series the relative RMSSE of smooth TBATS to it is computed on, and those left out because the RMSSE of smooth TBATS or of the comparator is zero.", "conditions": { "method": "D1" }, "row_labels": [ "comparator" ], "columns": [ { "variable": "series in relative RMSSE", "statistic": "count" }, { "variable": "series excluded by zero RMSSE", "statistic": "count" } ], "rows": [ { "id": "r1", "label": [ "D3" ], "values": [ 369, 0 ] }, { "id": "r2", "label": [ "D4" ], "values": [ 369, 0 ] }, { "id": "r3", "label": [ "D5" ], "values": [ 369, 0 ] }, { "id": "r4", "label": [ "D6" ], "values": [ 368, 1 ] } ], "artefact": "A8" }, { "id": "T3", "description": "The one-sided t-tests of the plan on the per-series log ratios of smooth TBATS to the comparator: p-value and p-value adjusted by Holm over the four tests.", "row_labels": [ "comparator", "measure" ], "columns": [ { "variable": "p-value, one-sided t-test", "statistic": "p_value" }, { "variable": "Holm-adjusted p-value, one-sided t-test", "statistic": "p_value" } ], "rows": [ { "id": "r1", "label": [ "D3", "RMSSE" ], "values": [ 0.01407, 0.02814260346047013 ] }, { "id": "r2", "label": [ "D4", "RMSSE" ], "values": [ 0.06692, 0.06691757930915539 ] }, { "id": "r3", "label": [ "D6", "RMSSE" ], "values": [ 4.861e-42, 1.4582738129581806e-41 ] }, { "id": "r4", "label": [ "D3", "elapsed time" ], "values": [ 3.609e-85, 1.444e-84 ] } ], "artefact": "A8" } ], "artefacts": [ "A8", "A10" ], "lab_runs": [ "RN13" ], "setup": { "studied": [ "D1", "D3", "D4", "D5", "D6" ], "cases": [ "D18" ], "procedure": [ "D19", "D15" ], "lens": [ "D16", "D17", "D12", "D13" ] }, "execution": { "code_version": "7cd242018acdf360e5fb49b3d990b72eb5395298", "software": { "python": "3.14.4", "platform": "Linux-7.0.0-42-generic-x86_64-with-glibc2.43" } }, "availability": { "level": "public" }, "data_origin": "secondary" }, { "id": "E5", "profile": "quant", "description": "[[D20:smooth TBATS dnorm]] against [[D1:smooth TBATS]] (distribution \"auto\") on the 370 series of [[D18:electricity hourly]] with [[D19:rolling-origin evaluation]] (the windows of run RN13), run once under plan PL8 (lab run RN14). Both are fitted to each window one after the other in the same single-threaded worker, so that their elapsed times are measured under the same load; a failed fit gets the seasonal naive 168 forecasts and no interval. Per series, the RMSSE, the coverage of the 95% interval and the elapsed time averaged over the 7 origins; one-sided t-tests over series on the mean log ratio of RMSSE (auto over dnorm), the mean coverage of dnorm minus 0.95 and the mean log ratio of elapsed time (dnorm over auto), adjusted by Holm over the three; the [[D15:Nemenyi test]] on the six methods (A1). Series 322 has an undefined RMSSE and is left out.", "results": [ { "id": "R1", "variable": "D17", "value": 0.9274901866520837, "interval": { "level": 0.99, "lower": 0.782479221333724, "upper": 1.099374939144907, "kind": "confidence" }, "conditions": { "method": "D20", "comparator": "D1" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R2", "variable": "relative elapsed time", "value": 0.636836353873358, "interval": { "level": 0.99, "lower": 0.6189769713949341, "upper": 0.6552110342663261, "kind": "confidence" }, "conditions": { "method": "D20", "comparator": "D1" }, "statistic": { "name": "mean", "kind": "geometric" } }, { "id": "R3", "variable": "coverage of smooth TBATS dnorm minus 0.95", "value": -0.01679894179894176, "interval": { "level": 0.99, "lower": -0.02364168981181017, "upper": -0.00995619378607335, "kind": "confidence" }, "statistic": "mean" }, { "id": "R4", "variable": "log ratio of RMSSE, smooth TBATS to smooth TBATS dnorm", "value": 0.07527306498554022, "interval": { "level": 0.99, "lower": -0.09474178115542094, "upper": 0.2452879111265014, "kind": "confidence" }, "statistic": "mean" }, { "id": "R5", "variable": "log ratio of elapsed time, smooth TBATS dnorm to smooth TBATS", "value": -0.4512425577124759, "interval": { "level": 0.99, "lower": -0.47968720990568653, "upper": -0.42279790551926527, "kind": "confidence" }, "statistic": "mean" }, { "id": "R6", "variable": "critical distance, Nemenyi test (RMSSE)", "value": 0.46329573710090965, "conditions": { "level": 0.99 } }, { "id": "R7", "variable": "series in Nemenyi test", "value": 369, "statistic": "count" }, { "id": "R8", "variable": "p-value, rmcb test of equal mean ranks (RMSSE)", "value": 1.0372190587495083e-94, "statistic": "p_value" }, { "id": "R9", "variable": "series excluded by undefined RMSSE", "value": 1, "statistic": "count" } ], "tables": [ { "id": "T1", "description": "Per method: mean and median RMSSE over series, coverage of the 95% prediction intervals, median elapsed time per series in seconds, and the series and fits that failed.", "row_labels": [ "method" ], "columns": [ { "variable": "D16", "statistic": "mean" }, { "variable": "D16", "statistic": "median" }, { "variable": "D12", "conditions": { "nominal level": 0.95 }, "statistic": "mean" }, { "variable": "D13", "statistic": "median" }, { "variable": "series failed", "statistic": "count" }, { "variable": "fits failed", "statistic": "count" } ], "rows": [ { "id": "r1", "label": [ "D1" ], "values": [ 0.7948, 0.6644, 0.954, 12.64, 0, 0 ] }, { "id": "r2", "label": [ "D20" ], "values": [ 0.7836, 0.6554, 0.9332, 8.031, 0, 0 ] } ], "artefact": "A8" }, { "id": "T2", "description": "Nemenyi test on the per-series RMSSE of the six methods: smooth TBATS and smooth TBATS dnorm from this run, the other four from run RN13 on the same windows; mean rank with the bounds of its 99% interval.", "row_labels": [ "method" ], "columns": [ { "variable": "mean rank of RMSSE, Nemenyi test", "statistic": "mean" }, { "variable": "mean rank of RMSSE, Nemenyi test", "conditions": { "bound": "99% interval, lower" } }, { "variable": "mean rank of RMSSE, Nemenyi test", "conditions": { "bound": "99% interval, upper" } } ], "rows": [ { "id": "r1", "label": [ "D1" ], "values": [ 3.03, 2.798, 3.261 ] }, { "id": "r2", "label": [ "D3" ], "values": [ 3.122, 2.89, 3.354 ] }, { "id": "r3", "label": [ "D4" ], "values": [ 2.921, 2.69, 3.153 ] }, { "id": "r4", "label": [ "D5" ], "values": [ 3.995, 3.763, 4.226 ] }, { "id": "r5", "label": [ "D6" ], "values": [ 5.165, 4.934, 5.397 ] }, { "id": "r6", "label": [ "D20" ], "values": [ 2.767, 2.535, 2.999 ] } ], "artefact": "A4" }, { "id": "T3", "description": "The three one-sided t-tests of plan PL8 over series: p-value, p-value adjusted by Holm over the three, series in the test and series left out for a non-finite value.", "row_labels": [ "measure" ], "columns": [ { "variable": "p-value, one-sided t-test", "statistic": "p_value" }, { "variable": "Holm-adjusted p-value, one-sided t-test", "statistic": "p_value" }, { "variable": "series in the t-test", "statistic": "count" }, { "variable": "series excluded by a non-finite value", "statistic": "count" } ], "rows": [ { "id": "r1", "label": [ "RMSSE" ], "values": [ 0.8738, 0.8738080346463214, 369, 0 ] }, { "id": "r2", "label": [ "coverage" ], "values": [ 3.054e-10, 6.107568971470071e-10, 369, 0 ] }, { "id": "r3", "label": [ "elapsed time" ], "values": [ 8.144e-140, 2.4431847468486607e-139, 369, 0 ] } ], "artefact": "A3" } ], "artefacts": [ "A1", "A2", "A3", "A4", "A8", "A10" ], "lab_runs": [ "RN14" ], "derived_from": [ { "evidence": "E4", "how": "reanalysis" } ], "setup": { "studied": [ "D20", "D1" ], "cases": [ "D18" ], "procedure": [ "D19" ], "lens": [ "D16", "D17", "D12", "D13" ] }, "execution": { "code_version": "cfc2f285c07bfc9ec05809902dcc354a883f6cab", "software": { "python": "3.14.4", "platform": "Linux-7.0.0-42-generic-x86_64-with-glibc2.43" } }, "availability": { "level": "public" }, "data_origin": "secondary" } ], "artefacts": [ { "id": "A1", "kind": "figure", "uri": "https://entrylib.org/p/2026.000007/v1/files/files/nemenyi_electricity_six_methods.png", "availability": { "level": "public" }, "version": "sha256:0d4fda83435b81dbce46394e8ecd6eb015de748e39cfd3e21530664f056a3d85", "description": "Nemenyi test on the per-series RMSSE of the six methods on the electricity data (369 series, level 0.99): mean rank of each method with its 99% interval, from run RN14 (smooth TBATS, smooth TBATS dnorm) and run RN13 (the other four).", "function": "data_plot", "licence": "CC-BY-4.0", "shows": [ "E5.T2.r1", "E5.T2.r2", "E5.T2.r3", "E5.T2.r4", "E5.T2.r5", "E5.T2.r6" ] }, { "id": "A2", "kind": "table", "uri": "https://entrylib.org/p/2026.000007/v1/files/files/dnorm_methods.csv", "availability": { "level": "public" }, "version": "sha256:4fb964bbf3aa381b300bdbc37951c6c1ea25690483b66fc6c50d406a1974a462", "description": "CSV written by run RN14 with the per-method values of smooth TBATS and smooth TBATS dnorm (E5.T1), at full precision.", "licence": "CC-BY-4.0" }, { "id": "A3", "kind": "table", "uri": "https://entrylib.org/p/2026.000007/v1/files/files/dnorm_tests.csv", "availability": { "level": "public" }, "version": "sha256:13e708c3aefebd395edf371d2905624d7334bd7011b3c56b82e8f4ee5e720f15", "description": "CSV written by run RN14 with the three t-tests of plan PL8 and the relative RMSSE and elapsed time with their intervals (E5.T3, E5.R1 to E5.R5), at full precision.", "licence": "CC-BY-4.0" }, { "id": "A4", "kind": "table", "uri": "https://entrylib.org/p/2026.000007/v1/files/files/nemenyi_electricity_six_methods.csv", "availability": { "level": "public" }, "version": "sha256:8a8739d90b11f0c4673e3d7b5ee2063918779bbb9391cb62ac16ed5e99a1d983", "description": "CSV written by run RN14 with the Nemenyi test of the six methods: mean ranks, intervals, critical distance (E5.T2, E5.R6 to E5.R8), at full precision.", "licence": "CC-BY-4.0" }, { "id": "A8", "kind": "data", "uri": "https://entrylib.org/p/2026.000007/v1/files/files/smooth-tbats.results.csv", "version": "sha256:ae12e94c62124549f3b86e89845c3649871a16333ea641e4735f47be23eb16a9", "availability": { "level": "public" }, "description": "Every result of the lab record (1308 results of 14 runs) at full precision: the lab result's ID, its run, the quantity as the run's script named it, its conditions and its value (both as JSON). The evidence's tables give the values to four significant digits.", "columns": [ { "name": "lab_result", "dimension": "the result event of the lab record" }, { "name": "run", "dimension": "the lab run that produced it" }, { "name": "quantity", "dimension": "the quantity, as the run's script named it" }, { "name": "conditions", "dimension": "the result's conditions, as JSON" }, { "name": "value", "variable": "the value of the quantity, as JSON, at full precision" } ], "licence": "CC-BY-4.0" }, { "id": "A10", "kind": "code", "licence": "LGPL-2.1-only", "availability": { "level": "public" }, "uri": "https://github.com/openforecast-org/smooth/tree/cfc2f285c07bfc9ec05809902dcc354a883f6cab/claimset-code/smooth-tbats", "description": "The study's scripts: formulation_checks.py (E1), m4_hourly.py and m4_forecast_tbats.R (E2, E3), electricity_hourly.py (E4) and electricity_dnorm.py (E5), at the last commit of the study's code; each evidence names the commit it ran at." } ], "references": [ { "id": "B4", "target": "10.1198/jasa.2011.tm09771", "id_type": "doi" }, { "id": "B5", "target": "10.18637/jss.v027.i03", "id_type": "doi" }, { "id": "B6", "target": "10.1016/j.ijforecast.2019.04.014", "id_type": "doi" }, { "id": "B7", "target": "10.1016/j.ijforecast.2021.11.013", "id_type": "doi" }, { "id": "B8", "target": "10.1201/9781003452652", "id_type": "doi" }, { "id": "B9", "target": "10.24432/C58C86", "id_type": "doi" }, { "id": "B10", "target": "10.1016/j.ijforecast.2006.03.001", "id_type": "doi" }, { "id": "B11", "target": "https://jmlr.org/papers/v7/demsar06a.html", "id_type": "url", "archive_uri": "https://web.archive.org/web/20261031040308/https://www.jmlr.org/papers/v7/demsar06a.html" }, { "id": "B12", "target": "https://github.com/Nixtla/statsforecast", "id_type": "url", "archive_uri": "https://web.archive.org/web/20260624224307/https://github.com/Nixtla/statsforecast" } ] }