Claim C5 · descriptive · entrylib:2026.000010@v1 · 2026-11-16

Online Detection of Forecast Model Inadequacies Using Forecast Errors · C5

Principal claim C5 of this paper (Svetunkov, I. (guarantor), 2026), with what it rests on. Read the whole paper.

Text file for AI

Text file: the claim with what it rests on as plain text, to attach when your assistant cannot open the address.

The message

Read https://entrylib.org/p/2026.000010/C5, one claim of a research paper with the evidence it rests on. Tell me briefly what it says and how well it is supported. If you cannot open the link, tell me instead of guessing. Then wait for my questions.

The claim

SupportS 71C5 · descriptive · novelty: new, by the authors

On the seasonal mean change scenario, the seasonal trend change scenario and the autoregressive parameter change scenarios, monitoring forecast errors with the forecast error monitoring framework detected the changes with a shorter average detection delay (ADD) than Raw CUSUM, at a low false detection probability (FDP).

Claims it rests on

SupportS 86C8 · descriptive · novelty: new, by the authors

On the seasonal mean change scenario, ARMA Forecast Errors has a high detection probability (DP) and a lower average detection delay (ADD) than Raw CUSUM across all change sizes.

SupportS 86C9 · descriptive · novelty: new, by the authors

On the seasonal trend change scenario, ARMA Forecast Errors and Raw CUSUM both have an extremely high detection probability (DP), and ARMA Forecast Errors has the lower average detection delay (ADD) across all change sizes.

SupportS 78C10 · descriptive · novelty: new, by the authors

On the autoregressive parameter change scenarios, ARMA Forecast Errors and ETS Forecast Errors each give a better detection probability (DP) and average detection delay (ADD) than Raw CUSUM in all scenarios.

SupportS 89C11 · descriptive

On the autoregressive parameter change scenarios, the false detection probability (FDP) is below 0.075 for each of the compared detectors.

SupportS 87C12 · descriptive

On the seasonal mean change scenario and the seasonal trend change scenario, the false detection probability (FDP) of the compared detectors is low across all change sizes.

Evidence

E5 · simulation · cited by C5, C8, C9, C12

Simulation of a change in mean and a change in trend in seasonal data (Section 4.1 and Figure 3 of the paper): detection probability, false detection probability and average detection delay of the three detectors, 1000 repetitions per change size. The paper reports the values as a figure only.

E6 · simulation · cited by C5, C10, C11

Simulation of a change in the autoregressive parameter of AR(1) errors in quarterly seasonal data (Section 4.2 and Figure 4 of the paper), for pre-change parameters 0.2 and 0.8: detection probability, false detection probability and average detection delay of the three detectors, 1000 repetitions per scenario. The paper reports the values as a figure only. The paper also notes that detection probability and delay are better for all three detectors when the dependence increases after the change.

Limitations

L2 · model specification · could bias magnitude · affects C5, C8, C9, C10, C11, C12 · mitigation: none

The two forecasting-model assumptions that make the thresholds theoretically valid are not necessarily satisfied in all the simulated scenarios.

L3 · sampling · reduces transferability · affects C5, C8, C9, C10 · mitigation: none

All simulated data have an ARMA error structure with normally distributed innovations and a training sample of 300 observations.

L4 · measurement · reduces precision · affects C6 · mitigation: none

The start of the Christmas peak season at the delivery office is inferred from the rise in the mean and variance of the forecast errors; the underlying parcel volumes and forecasts are not known to the authors.

Definitions

Methods 8

D1 forecast error monitoring framework: A framework that monitors a forecasting model online by applying sequential changepoint detection to its one-step-ahead forecast errors. For a univariate series YtY_t and a forecasting model ℱ\mathcal{F} with one-step-ahead point forecast y^t(1)=𝔼ℱ[Yt|yt−1,…,y1]\hat{y}_t(1) = \mathbb{E}_{\mathcal{F}}[Y_t | y_{t-1}, \ldots, y_1], the forecast errors are et=Yt−y^t(1).e_t = Y_t - \hat{y}_t(1). The first mm forecast errors are the Phase I (training) sample; monitoring starts at t=m+1t = m + 1 and stops when a detector, the mean detector or the squared-error detector, crosses its threshold. Any forecasting approach can supply the errors: model-based, machine learning or expert judgement.

D2 · part of D1 mean detector $D_\mu(m,k)$: Page's CUSUM detector applied to the forecast errors of the forecast error monitoring framework: Qμ(m,k)=∑t=m+1m+ket−km∑t=1metDμ(m,k)=max0≤i≤k⁡|Qμ(m,k)−Qμ(m,i)|\begin{aligned} Q_\mu(m,k) &= \sum_{t=m+1}^{m+k} e_t - \frac{k}{m} \sum_{t=1}^{m} e_t \\ D_\mu(m,k) &= \max_{0 \le i \le k} \left| Q_\mu(m,k) - Q_\mu(m,i) \right| \end{aligned} with stopping time τμ,m=min⁡{k≥1:Dμ(m,k)≥σ^μ,mcμ,αg(m,k,γ)},\tau_{\mu,m} = \min\{k \ge 1 : D_\mu(m,k) \ge \hat{\sigma}_{\mu,m} c_{\mu,\alpha} g(m,k,\gamma)\}, where σ^μ,m\hat{\sigma}_{\mu,m} is the standard deviation of the forecast errors in the Phase I sample. The weight function g(m,k,γ)g(m,k,\gamma) is that of Page's CUSUM detector, and the critical constant cμ,αc_{\mu,\alpha} comes from the limit distribution of the detector under the null hypothesis. Source: PAGE (1954).

D3 · part of D1 squared-error detector $D_\xi(m,k)$: Page's CUSUM detector applied to the centred squared forecast errors of the forecast error monitoring framework: Qξ(m,k)=∑t=m+1m+k(et−b^μ,m)2−km∑t=1m(et−b^μ,m)2Dξ(m,k)=max0≤i≤k⁡|Qξ(m,k)−Qξ(m,i)|\begin{aligned} Q_\xi(m,k) &= \sum_{t=m+1}^{m+k} (e_t - \hat{b}_{\mu,m})^2 - \frac{k}{m} \sum_{t=1}^{m} (e_t - \hat{b}_{\mu,m})^2 \\ D_\xi(m,k) &= \max_{0 \le i \le k} \left| Q_\xi(m,k) - Q_\xi(m,i) \right| \end{aligned} with stopping time τξ,m=min⁡{k≥1:Dξ(m,k)≥σ^ξ,mcξ,αg(m,k,γ)},\tau_{\xi,m} = \min\{k \ge 1 : D_\xi(m,k) \ge \hat{\sigma}_{\xi,m} c_{\xi,\alpha} g(m,k,\gamma)\}, where b^μ,m\hat{b}_{\mu,m} is the mean of the forecast errors in the mm Phase I samples and σ^ξ,m\hat{\sigma}_{\xi,m} the standard deviation of the centred squared forecast errors in that sample. g(m,k,γ)g(m,k,\gamma) is that of Page's CUSUM detector and cξ,αc_{\xi,\alpha} comes from the limit distribution of the detector under the null hypothesis. The paper prints the second sum of Qξ(m,k)Q_\xi(m,k) without the square, a typographical error that the guarantor, an author of the paper, has confirmed.

D5 · a kind of D1 ARMA Forecast Errors: The detector of the forecast error monitoring framework, [[D2:Dμ(m,k)D_\mu(m,k)]] or [[D3:Dξ(m,k)D_\xi(m,k)]] by the type of change, applied to the forecast errors of an ARMA forecasting model fitted with the forecast R package. For seasonal data the model is a seasonal ARMA model chosen by auto.arima using AIC, with no differencing. The standard deviation is the traditional estimator, the errors being taken as i.i.d. Source: https://CRAN.R-project.org/package=forecast.

D6 · a kind of D1 ETS Forecast Errors: The detector of the forecast error monitoring framework, [[D2:Dμ(m,k)D_\mu(m,k)]] or [[D3:Dξ(m,k)D_\xi(m,k)]] by the type of change, applied to the forecast errors of an ETS forecasting model fitted with the es function of the smooth R package. The standard deviation is the traditional estimator, the errors being taken as i.i.d. Source: https://CRAN.R-project.org/package=smooth.

D4 Raw CUSUM: Page's CUSUM detector applied to the raw data instead of forecast errors; for variance changes, to the centred squared raw data. In the paper's simulations its standard deviation σ^m\hat{\sigma}_m is the Bartlett long run variance estimator from the sandwich R package. Source: Inclán and Tiao (1994).

D7 compared detectors: The three detectors compared in the paper's simulations. Members: D4, D5, D6.

D24 Page's CUSUM detector $D(m,k)$: The sequential detector of a mean change in a series YtY_t that follows the change in mean model Yt={λ+ϵt,t=1,…,m+k*λ+ϵt+Δm,t=m+k*+1,m+k*+2,…Y_t = \begin{cases} \lambda + \epsilon_t, & t = 1, \ldots, m + k^* \\ \lambda + \epsilon_t + \Delta_m, & t = m + k^* + 1, m + k^* + 2, \ldots \end{cases} with zero-mean variables ϵt\epsilon_t, a training period of length mm without changepoints and a change of size Δm\Delta_m at time k*k^* after it. With Q(m,k)=∑t=m+1m+kYt−km∑t=1mYtD(m,k)=max0≤i≤k⁡|Q(m,k)−Q(m,i)|\begin{aligned} Q(m,k) &= \sum_{t=m+1}^{m+k} Y_t - \frac{k}{m} \sum_{t=1}^{m} Y_t \\ D(m,k) &= \max_{0 \le i \le k} \left| Q(m,k) - Q(m,i) \right| \end{aligned} the null hypothesis H0:Δm=0H_0: \Delta_m = 0 is rejected in favour of HA:Δm≠0H_A: \Delta_m \ne 0 at the stopping time τm=min⁡{k≥1:D(m,k)≥σ^mcαg(m,k,γ)},\tau_m = \min\{k \ge 1 : D(m,k) \ge \hat{\sigma}_m c_\alpha g(m,k,\gamma)\}, where σ^m\hat{\sigma}_m is a weakly consistent estimator, from the mm training samples, of the constant σ\sigma of the weak invariance principle, the weight function is g(m,k,γ)=m(1+km)(km+k)γ,γ∈[0,1/2).g(m,k,\gamma) = \sqrt{m}\left(1 + \frac{k}{m}\right)\left(\frac{k}{m+k}\right)^{\gamma}, \quad \gamma \in [0, 1/2). and the critical value cαc_\alpha is taken from the limit distribution under H0H_0, so that limm→∞⁡P(τm<∞)=α\lim_{m\to\infty} P(\tau_m < \infty) = \alpha under H0H_0 for the chosen false alarm rate α\alpha and limm→∞⁡P(τm<∞)=1\lim_{m\to\infty} P(\tau_m < \infty) = 1 under HAH_A. All limits are for m→∞m \to \infty; the model is trained once and not re-trained during monitoring. The paper takes γ=0\gamma = 0. Source: Chu et al. (1996).

Data 3

D16 seasonal mean change scenario: Simulated data whose errors follow an ARMA(2,1) model with ϕ=(−0.6,0.3)\phi = (-0.6, 0.3), θ=−0.3\theta = -0.3, λ=0\lambda = 0 and σ2=1\sigma^2 = 1, with a mean change of size δμ\delta_\mu at k*=100k^* = 100 and a seasonal component of frequency 12 whose seasonal means are 12 equally spaced points of y=10sin⁡(x)y = 10\sin(x), x∈[0,π]x \in [0, \pi].

D17 seasonal trend change scenario: Simulated data yt=1[t>m+k*]βt+ϵty_t = 1[t > m + k^*]\beta t + \epsilon_t whose errors follow an ARMA(1,1) model with ϕ=0.2\phi = 0.2, θ=0.2\theta = 0.2, λ=0\lambda = 0 and σ2=1\sigma^2 = 1: no trend before the change and a trend of gradient β\beta after it, with a seasonal component of frequency 7 whose seasonal means are 7 equally spaced points of y=10cos⁡(x)y = 10\cos(x), x∈[0,2π]x \in [0, 2\pi].

D18 autoregressive parameter change scenarios: Simulated data with AR(1) errors, λ=0\lambda = 0, whose autoregressive parameter changes at the changepoint from ϕpre=0.2\phi_{\text{pre}} = 0.2 (scenario 1) or ϕpre=0.8\phi_{\text{pre}} = 0.8 (scenario 2) to varying values ϕpost\phi_{\text{post}}, with quarterly seasonality given by quarter means (−2,5,7,−10)(-2, 5, 7, -10). The ETS forecasting model is ETS(A,N,A) with estimated smoothing parameters.

Variables and measures 3

D12 average detection delay (ADD): The mean of the detection delays over the simulated series in which a change was signalled within the length of the data.

D13 detection probability (DP): The proportion of simulated series in which a change was detected within the length of the monitoring data.

D14 false detection probability (FDP): The proportion of simulated series in which a change was detected before the true change had occurred.

Study design 1

D15 simulation design of the paper: Each scenario is simulated 1000 times with normally distributed innovations. Forecasting models are trained on the first ntrain=300n_{\text{train}} = 300 observations and then produce one-step-ahead forecasts sequentially. A change is signalled by the theoretical thresholds of Page's CUSUM detector with γ=0\gamma = 0 and a false alarm rate of 0.05.

Context

Addresses B5 · from C5
The closest result found: Robbins et al. (2011) showed, for offline changepoint detection, that using residuals within the CUSUM procedure improves on the raw data under temporal dependence; the claim concerns sequential detection with forecast errors of any forecasting model.
Addresses B5 · from C8
Robbins et al. (2011) compared residuals with raw data for offline mean shift testing; this is the sequential case.
Addresses B6 · from C9
Horváth et al. (2004) monitor changes in linear models, including trend, within a specified model; here the trend change is detected through forecast errors.
Addresses B1 · from C10
Aue et al. (2015) detect changes in ARMA parameters from ARMA residuals; here the comparison is with monitoring the raw data, and includes an ETS forecasting model.

References

  1. Chu, C.-S. J., Stinchcombe, M., White, H., 1996. Monitoring Structural Change. Econometrica 64, 1045. https://doi.org/10.2307/2171955 Used by: D24
  2. Grundy, T., Killick, R., Svetunkov, I., 2026. Online Detection of Forecast Model Inadequacies Using Forecast Errors. Journal of Time Series Analysis 47, 715–726. https://doi.org/10.1111/jtsa.12843 Used by: A3, A4
  3. Inclán, C., Tiao, G. C., 1994. Use of Cumulative Sums of Squares for Retrospective Detection of Changes of Variance. Journal of the American Statistical Association 89, 913–923. https://doi.org/10.1080/01621459.1994.10476824 Used by: D4
  4. PAGE, E. S., 1954. CONTINUOUS INSPECTION SCHEMES. Biometrika 41, 100–115. https://doi.org/10.1093/biomet/41.1-2.100 Used by: D2
  5. https://CRAN.R-project.org/package=forecast Used by: D5, A8
  6. https://CRAN.R-project.org/package=sandwich Used by: A10
  7. https://CRAN.R-project.org/package=smooth Used by: D6, A9

Artefacts

  • A3 · figure · data_plot · Grundy et al. (2026) Detection probability, false detection probability and average detection delay of the three detectors for a change in mean (top) and a change in trend (bottom) in seasonal data.
  • A4 · figure · data_plot · Grundy et al. (2026) Detection probability, false detection probability and average detection delay of the three detectors for a change in the autoregressive parameter, pre-change parameter 0.2 (top) and 0.8 (bottom).
  • A8 · software · https://CRAN.R-project.org/package=forecast The forecast R package (Arima and auto.arima), used for the ARMA forecasting models. The paper's reference gives no version; 8.21.1 is the last version released in 2023, when the paper was written. package=forecast
  • A9 · software · https://CRAN.R-project.org/package=smooth The smooth R package (es function), used for the ETS forecasting models, in the version the paper's reference states. package=smooth
  • A10 · software · https://CRAN.R-project.org/package=sandwich The sandwich R package, used for the Bartlett long run variance estimator of Raw CUSUM. The paper's reference gives no version; 3.1-0 is the last version released in 2023, when the paper was written. package=sandwich

Score

Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.

C5: 71, Partly supported

Novelty: new, by the authors

  1. −10 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
  2. −7 Checkable Flags: availability not stated, figure without data. To remove it: Publish what each flag says is missing.
  3. −6 Open problems Limitation L2 (could bias magnitude, not mitigated): The two forecasting-model assumptions that make the thresholds theoretically valid are not necessarily satisfied in all the simulated scenarios. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  4. −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
  5. −1 Open problems Limitation L3 (reduces transferability, not mitigated): All simulated data have an ARMA error structure with normally distributed innovations and a training sample of 300 observations. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C8: 86, Well supported

Novelty: new, by the authors

  1. −7 Checkable Flags: availability not stated, figure without data. To remove it: Publish what each flag says is missing.
  2. −6 Open problems Limitation L2 (could bias magnitude, not mitigated): The two forecasting-model assumptions that make the thresholds theoretically valid are not necessarily satisfied in all the simulated scenarios. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −1 Open problems Limitation L3 (reduces transferability, not mitigated): All simulated data have an ARMA error structure with normally distributed innovations and a training sample of 300 observations. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C9: 86, Well supported

Novelty: new, by the authors

  1. −7 Checkable Flags: availability not stated, figure without data. To remove it: Publish what each flag says is missing.
  2. −6 Open problems Limitation L2 (could bias magnitude, not mitigated): The two forecasting-model assumptions that make the thresholds theoretically valid are not necessarily satisfied in all the simulated scenarios. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −1 Open problems Limitation L3 (reduces transferability, not mitigated): All simulated data have an ARMA error structure with normally distributed innovations and a training sample of 300 observations. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C10: 78, Partly supported

Novelty: new, by the authors

  1. −10 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
  2. −6 Open problems Limitation L2 (could bias magnitude, not mitigated): The two forecasting-model assumptions that make the thresholds theoretically valid are not necessarily satisfied in all the simulated scenarios. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −5 Checkable Flags: availability not stated, figure without data. To remove it: Publish what each flag says is missing.
  4. −1 Open problems Limitation L3 (reduces transferability, not mitigated): All simulated data have an ARMA error structure with normally distributed innovations and a training sample of 300 observations. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C11: 89, Well supported

  1. −6 Open problems Limitation L2 (could bias magnitude, not mitigated): The two forecasting-model assumptions that make the thresholds theoretically valid are not necessarily satisfied in all the simulated scenarios. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −5 Checkable Flags: availability not stated, figure without data. To remove it: Publish what each flag says is missing.

C12: 87, Well supported

  1. −7 Checkable Flags: availability not stated, figure without data. To remove it: Publish what each flag says is missing.
  2. −6 Open problems Limitation L2 (could bias magnitude, not mitigated): The two forecasting-model assumptions that make the thresholds theoretically valid are not necessarily satisfied in all the simulated scenarios. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.