CLAIM C5 of entrylib:2026.000010 Online Detection of Forecast Model Inadequacies Using Forecast Errors licence CC-BY-4.0; contributions C5 DEFINITIONS D1 [method] forecast error monitoring framework: A framework that monitors a forecasting model online by applying sequential changepoint detection to its one-step-ahead forecast errors. For a univariate series $Y_t$ and a forecasting model $\mathcal{F}$ with one-step-ahead point forecast $\hat{y}_t(1) = \mathbb{E}_{\mathcal{F}}[Y_t | y_{t-1}, \ldots, y_1]$, the forecast errors are $$e_t = Y_t - \hat{y}_t(1).$$ The first $m$ forecast errors are the Phase I (training) sample; monitoring starts at $t = m + 1$ and stops when a detector, [[D2:the mean detector]] or [[D3:the squared-error detector]], crosses its threshold. Any forecasting approach can supply the errors: model-based, machine learning or expert judgement. D2 [method] mean detector $D_\mu(m,k)$: [[D24:Page's CUSUM detector]] applied to the forecast errors of the [[D1:forecast error monitoring framework]]: $$\begin{aligned} Q_\mu(m,k) &= \sum_{t=m+1}^{m+k} e_t - \frac{k}{m} \sum_{t=1}^{m} e_t \\ D_\mu(m,k) &= \max_{0 \le i \le k} \left| Q_\mu(m,k) - Q_\mu(m,i) \right| \end{aligned}$$ with stopping time $$\tau_{\mu,m} = \min\{k \ge 1 : D_\mu(m,k) \ge \hat{\sigma}_{\mu,m} c_{\mu,\alpha} g(m,k,\gamma)\},$$ where $\hat{\sigma}_{\mu,m}$ is the standard deviation of the forecast errors in the Phase I sample. The weight function $g(m,k,\gamma)$ is that of [[D24:Page's CUSUM detector]], and the critical constant $c_{\mu,\alpha}$ comes from the limit distribution of the detector under the null hypothesis. [part of D1] [source B11] D3 [method] squared-error detector $D_\xi(m,k)$: [[D24:Page's CUSUM detector]] applied to the centred squared forecast errors of the [[D1:forecast error monitoring framework]]: $$\begin{aligned} Q_\xi(m,k) &= \sum_{t=m+1}^{m+k} (e_t - \hat{b}_{\mu,m})^2 - \frac{k}{m} \sum_{t=1}^{m} (e_t - \hat{b}_{\mu,m})^2 \\ D_\xi(m,k) &= \max_{0 \le i \le k} \left| Q_\xi(m,k) - Q_\xi(m,i) \right| \end{aligned}$$ with stopping time $$\tau_{\xi,m} = \min\{k \ge 1 : D_\xi(m,k) \ge \hat{\sigma}_{\xi,m} c_{\xi,\alpha} g(m,k,\gamma)\},$$ where $\hat{b}_{\mu,m}$ is the mean of the forecast errors in the $m$ Phase I samples and $\hat{\sigma}_{\xi,m}$ the standard deviation of the centred squared forecast errors in that sample. $g(m,k,\gamma)$ is that of [[D24:Page's CUSUM detector]] and $c_{\xi,\alpha}$ comes from the limit distribution of the detector under the null hypothesis. The paper prints the second sum of $Q_\xi(m,k)$ without the square, a typographical error that the guarantor, an author of the paper, has confirmed. [part of D1] D4 [method] Raw CUSUM: [[D24:Page's CUSUM detector]] applied to the raw data instead of forecast errors; for variance changes, to the centred squared raw data. In the paper's simulations its standard deviation $\hat{\sigma}_m$ is the Bartlett long run variance estimator from the sandwich R package. [source B2] D5 [method] ARMA Forecast Errors: The detector of the [[D1:forecast error monitoring framework]], [[D2:$D_\mu(m,k)$]] or [[D3:$D_\xi(m,k)$]] by the type of change, applied to the forecast errors of an ARMA forecasting model fitted with the forecast R package. For seasonal data the model is a seasonal ARMA model chosen by auto.arima using AIC, with no differencing. The standard deviation is the traditional estimator, the errors being taken as i.i.d. [a kind of D1] [source B12] D6 [method] ETS Forecast Errors: The detector of the [[D1:forecast error monitoring framework]], [[D2:$D_\mu(m,k)$]] or [[D3:$D_\xi(m,k)$]] by the type of change, applied to the forecast errors of an ETS forecasting model fitted with the es function of the smooth R package. The standard deviation is the traditional estimator, the errors being taken as i.i.d. [a kind of D1] [source B13] D7 [method] compared detectors: The three detectors compared in the paper's simulations. [members D4, D5, D6] D12 [variable] average detection delay (ADD): The mean of the detection delays over the simulated series in which a change was signalled within the length of the data. D13 [variable] detection probability (DP): The proportion of simulated series in which a change was detected within the length of the monitoring data. D14 [variable] false detection probability (FDP): The proportion of simulated series in which a change was detected before the true change had occurred. D15 [design] simulation design of the paper: Each scenario is simulated 1000 times with normally distributed innovations. Forecasting models are trained on the first $n_{\text{train}} = 300$ observations and then produce one-step-ahead forecasts sequentially. A change is signalled by the theoretical thresholds of [[D24:Page's CUSUM detector]] with $\gamma = 0$ and a false alarm rate of 0.05. D16 [data] seasonal mean change scenario: Simulated data whose errors follow an ARMA(2,1) model with $\phi = (-0.6, 0.3)$, $\theta = -0.3$, $\lambda = 0$ and $\sigma^2 = 1$, with a mean change of size $\delta_\mu$ at $k^* = 100$ and a seasonal component of frequency 12 whose seasonal means are 12 equally spaced points of $y = 10\sin(x)$, $x \in [0, \pi]$. D17 [data] seasonal trend change scenario: Simulated data $y_t = 1[t > m + k^*]\beta t + \epsilon_t$ whose errors follow an ARMA(1,1) model with $\phi = 0.2$, $\theta = 0.2$, $\lambda = 0$ and $\sigma^2 = 1$: no trend before the change and a trend of gradient $\beta$ after it, with a seasonal component of frequency 7 whose seasonal means are 7 equally spaced points of $y = 10\cos(x)$, $x \in [0, 2\pi]$. D18 [data] autoregressive parameter change scenarios: Simulated data with AR(1) errors, $\lambda = 0$, whose autoregressive parameter changes at the changepoint from $\phi_{\text{pre}} = 0.2$ (scenario 1) or $\phi_{\text{pre}} = 0.8$ (scenario 2) to varying values $\phi_{\text{post}}$, with quarterly seasonality given by quarter means $(-2, 5, 7, -10)$. The ETS forecasting model is ETS(A,N,A) with estimated smoothing parameters. D24 [method] Page's CUSUM detector $D(m,k)$: The sequential detector of a mean change in a series $Y_t$ that follows the change in mean model $$Y_t = \begin{cases} \lambda + \epsilon_t, & t = 1, \ldots, m + k^* \\ \lambda + \epsilon_t + \Delta_m, & t = m + k^* + 1, m + k^* + 2, \ldots \end{cases}$$ with zero-mean variables $\epsilon_t$, a training period of length $m$ without changepoints and a change of size $\Delta_m$ at time $k^*$ after it. With $$\begin{aligned} Q(m,k) &= \sum_{t=m+1}^{m+k} Y_t - \frac{k}{m} \sum_{t=1}^{m} Y_t \\ D(m,k) &= \max_{0 \le i \le k} \left| Q(m,k) - Q(m,i) \right| \end{aligned}$$ the null hypothesis $H_0: \Delta_m = 0$ is rejected in favour of $H_A: \Delta_m \ne 0$ at the stopping time $$\tau_m = \min\{k \ge 1 : D(m,k) \ge \hat{\sigma}_m c_\alpha g(m,k,\gamma)\},$$ where $\hat{\sigma}_m$ is a weakly consistent estimator, from the $m$ training samples, of the constant $\sigma$ of the weak invariance principle, the weight function is $$g(m,k,\gamma) = \sqrt{m}\left(1 + \frac{k}{m}\right)\left(\frac{k}{m+k}\right)^{\gamma}, \quad \gamma \in [0, 1/2).$$ and the critical value $c_\alpha$ is taken from the limit distribution under $H_0$, so that $\lim_{m\to\infty} P(\tau_m < \infty) = \alpha$ under $H_0$ for the chosen false alarm rate $\alpha$ and $\lim_{m\to\infty} P(\tau_m < \infty) = 1$ under $H_A$. All limits are for $m \to \infty$; the model is trained once and not re-trained during monitoring. The paper takes $\gamma = 0$. [source B10] CLAIMS C5 [descriptive; principal; support: simulation E5 artefacts A3, simulation E6 artefacts A4; uses C8, C9, C10, C11, C12; limitations L2, L3; strength moderate (author); verifiability open; planning exploratory] On [[D16:the seasonal mean change scenario]], [[D17:the seasonal trend change scenario]] and [[D18:the autoregressive parameter change scenarios]], monitoring forecast errors with the [[D1:forecast error monitoring framework]] detected the changes with a shorter [[D12:average detection delay (ADD)]] than [[D4:Raw CUSUM]], at a low [[D14:false detection probability (FDP)]]. scope: data_type: simulated seasonal time series with ARMA errors and normally distributed innovations | conditions: training sample of 300 observations; false alarm rate 0.05 addresses B5: The closest result found: Robbins et al. (2011) showed, for offline changepoint detection, that using residuals within the CUSUM procedure improves on the raw data under temporal dependence; the claim concerns sequential detection with forecast errors of any forecasting model. C8 [descriptive; supporting; support: simulation E5 artefacts A3; limitations L2, L3; strength moderate (author); verifiability open; planning exploratory] On [[D16:the seasonal mean change scenario]], [[D5:ARMA Forecast Errors]] has a high [[D13:detection probability (DP)]] and a lower [[D12:average detection delay (ADD)]] than [[D4:Raw CUSUM]] across all change sizes. scope: data_type: simulated monthly-type seasonal data with ARMA(2,1) errors | conditions: mean change sizes 0.5 to 2.5 addresses B5: Robbins et al. (2011) compared residuals with raw data for offline mean shift testing; this is the sequential case. C9 [descriptive; supporting; support: simulation E5 artefacts A3; limitations L2, L3; strength moderate (author); verifiability open; planning exploratory] On [[D17:the seasonal trend change scenario]], [[D5:ARMA Forecast Errors]] and [[D4:Raw CUSUM]] both have an extremely high [[D13:detection probability (DP)]], and [[D5:ARMA Forecast Errors]] has the lower [[D12:average detection delay (ADD)]] across all change sizes. scope: data_type: simulated daily-type seasonal data with ARMA(1,1) errors | conditions: trend gradients 0.008 to 0.042 addresses B6: Horváth et al. (2004) monitor changes in linear models, including trend, within a specified model; here the trend change is detected through forecast errors. C10 [descriptive; supporting; support: simulation E6 artefacts A4; limitations L2, L3; strength moderate (author); verifiability open; planning exploratory] On [[D18:the autoregressive parameter change scenarios]], [[D5:ARMA Forecast Errors]] and [[D6:ETS Forecast Errors]] each give a better [[D13:detection probability (DP)]] and [[D12:average detection delay (ADD)]] than [[D4:Raw CUSUM]] in all scenarios. scope: data_type: simulated quarterly-type seasonal data with AR(1) errors addresses B1: Aue et al. (2015) detect changes in ARMA parameters from ARMA residuals; here the comparison is with monitoring the raw data, and includes an ETS forecasting model. C11 [descriptive; supporting; not_applicable; support: simulation E6 (E6.R1) artefacts A4; limitations L2; strength moderate (author); verifiability open; planning exploratory] On [[D18:the autoregressive parameter change scenarios]], the [[D14:false detection probability (FDP)]] is below 0.075 for each of the [[D7:compared detectors]]. scope: data_type: simulated quarterly-type seasonal data with AR(1) errors | conditions: false alarm rate set at 0.05 C12 [descriptive; supporting; not_applicable; support: simulation E5 artefacts A3; limitations L2; strength moderate (author); verifiability open; planning exploratory] On [[D16:the seasonal mean change scenario]] and [[D17:the seasonal trend change scenario]], the [[D14:false detection probability (FDP)]] of the [[D7:compared detectors]] is low across all change sizes. scope: data_type: simulated seasonal data with ARMA errors EVIDENCE E5 [simulation; runs 1000] Simulation of a change in mean and a change in trend in seasonal data (Section 4.1 and Figure 3 of the paper): detection probability, false detection probability and average detection delay of the three detectors, 1000 repetitions per change size. The paper reports the values as a figure only. setup: studied D1, D7; cases D16, D17; procedure D15; lens D12, D13, D14 E6 [simulation; runs 1000] Simulation of a change in the autoregressive parameter of AR(1) errors in quarterly seasonal data (Section 4.2 and Figure 4 of the paper), for pre-change parameters 0.2 and 0.8: detection probability, false detection probability and average detection delay of the three detectors, 1000 repetitions per scenario. The paper reports the values as a figure only. The paper also notes that detection probability and delay are better for all three detectors when the dependence increases after the change. setup: studied D1, D7; cases D18; procedure D15; lens D12, D13, D14 E6.R1 D14 = < 0.075 (upper limit) 1 {D7: each detector, scenario: D18} LIMITATIONS L2 [model_specification; affects C5, C8, C9, C10, C11, C12; could_bias_magnitude; mitigation none] The two forecasting-model assumptions that make the thresholds theoretically valid are not necessarily satisfied in all the simulated scenarios. L3 [sampling; affects C5, C8, C9, C10; reduces_transferability; mitigation none] All simulated data have an ARMA error structure with normally distributed innovations and a training sample of 300 observations. L4 [measurement; affects C6; reduces_precision; mitigation none] The start of the Christmas peak season at the delivery office is inferred from the rise in the mean and variance of the forecast errors; the underlying parcel volumes and forecasts are not known to the authors. ARTEFACTS A3 [figure, data_plot] Detection probability, false detection probability and average detection delay of the three detectors for a change in mean (top) and a change in trend (bottom) in seasonal data. A4 [figure, data_plot] Detection probability, false detection probability and average detection delay of the three detectors for a change in the autoregressive parameter, pre-change parameter 0.2 (top) and 0.8 (bottom). shows E6.R1 A8 [software] The forecast R package (Arima and auto.arima), used for the ARMA forecasting models. The paper's reference gives no version; 8.21.1 is the last version released in 2023, when the paper was written. A9 [software] The smooth R package (es function), used for the ETS forecasting models, in the version the paper's reference states. A10 [software] The sandwich R package, used for the Bartlett long run variance estimator of Raw CUSUM. The paper's reference gives no version; 3.1-0 is the last version released in 2023, when the paper was written. REFERENCES B2 10.1080/01621459.1994.10476824 (doi) B7 10.1111/jtsa.12843 (doi) B10 10.2307/2171955 (doi) B11 10.1093/biomet/41.1-2.100 (doi) B12 https://CRAN.R-project.org/package=forecast (url) B13 https://CRAN.R-project.org/package=smooth (url) B14 https://CRAN.R-project.org/package=sandwich (url) HOW TO CITE This claim: Svetunkov, I. (guarantor) (2026). Claim C5, in: Online Detection of Forecast Model Inadequacies Using Forecast Errors. EntryLib. entrylib:2026.000010@v1#C5. https://entrylib.org/p/2026.000010/C5 The claimset it is from: Svetunkov, I. (guarantor) (2026). Online Detection of Forecast Model Inadequacies Using Forecast Errors. EntryLib. entrylib:2026.000010@v1. https://entrylib.org/p/2026.000010@v1