CLAIM C6 of entrylib:2026.000010 Online Detection of Forecast Model Inadequacies Using Forecast Errors licence CC-BY-4.0; contributions C6 DEFINITIONS D1 [method] forecast error monitoring framework: A framework that monitors a forecasting model online by applying sequential changepoint detection to its one-step-ahead forecast errors. For a univariate series $Y_t$ and a forecasting model $\mathcal{F}$ with one-step-ahead point forecast $\hat{y}_t(1) = \mathbb{E}_{\mathcal{F}}[Y_t | y_{t-1}, \ldots, y_1]$, the forecast errors are $$e_t = Y_t - \hat{y}_t(1).$$ The first $m$ forecast errors are the Phase I (training) sample; monitoring starts at $t = m + 1$ and stops when a detector, [[D2:the mean detector]] or [[D3:the squared-error detector]], crosses its threshold. Any forecasting approach can supply the errors: model-based, machine learning or expert judgement. D2 [method] mean detector $D_\mu(m,k)$: [[D24:Page's CUSUM detector]] applied to the forecast errors of the [[D1:forecast error monitoring framework]]: $$\begin{aligned} Q_\mu(m,k) &= \sum_{t=m+1}^{m+k} e_t - \frac{k}{m} \sum_{t=1}^{m} e_t \\ D_\mu(m,k) &= \max_{0 \le i \le k} \left| Q_\mu(m,k) - Q_\mu(m,i) \right| \end{aligned}$$ with stopping time $$\tau_{\mu,m} = \min\{k \ge 1 : D_\mu(m,k) \ge \hat{\sigma}_{\mu,m} c_{\mu,\alpha} g(m,k,\gamma)\},$$ where $\hat{\sigma}_{\mu,m}$ is the standard deviation of the forecast errors in the Phase I sample. The weight function $g(m,k,\gamma)$ is that of [[D24:Page's CUSUM detector]], and the critical constant $c_{\mu,\alpha}$ comes from the limit distribution of the detector under the null hypothesis. [part of D1] [source B11] D3 [method] squared-error detector $D_\xi(m,k)$: [[D24:Page's CUSUM detector]] applied to the centred squared forecast errors of the [[D1:forecast error monitoring framework]]: $$\begin{aligned} Q_\xi(m,k) &= \sum_{t=m+1}^{m+k} (e_t - \hat{b}_{\mu,m})^2 - \frac{k}{m} \sum_{t=1}^{m} (e_t - \hat{b}_{\mu,m})^2 \\ D_\xi(m,k) &= \max_{0 \le i \le k} \left| Q_\xi(m,k) - Q_\xi(m,i) \right| \end{aligned}$$ with stopping time $$\tau_{\xi,m} = \min\{k \ge 1 : D_\xi(m,k) \ge \hat{\sigma}_{\xi,m} c_{\xi,\alpha} g(m,k,\gamma)\},$$ where $\hat{b}_{\mu,m}$ is the mean of the forecast errors in the $m$ Phase I samples and $\hat{\sigma}_{\xi,m}$ the standard deviation of the centred squared forecast errors in that sample. $g(m,k,\gamma)$ is that of [[D24:Page's CUSUM detector]] and $c_{\xi,\alpha}$ comes from the limit distribution of the detector under the null hypothesis. The paper prints the second sum of $Q_\xi(m,k)$ without the square, a typographical error that the guarantor, an author of the paper, has confirmed. [part of D1] D8 [data] mean change model: A data-generating process with a changepoint at an unknown location $k^*$: $$\begin{aligned} Y_t | y_{t-1}, \ldots, y_1 &\sim F, \quad t = 1, \ldots, m + k^* \\ Y_t | y_{t-1}, \ldots, y_1 &\sim G, \quad t = m + k^* + 1, m + k^* + 2, \ldots \end{aligned}$$ where $F$ and $G$ differ in expectation by $\delta_{\mu,m} \ne 0$. The raw data may be temporally dependent, trended, seasonal or depend on exogenous variables. D9 [data] mean and variance change model: The [[D8:mean change model]] extended so that the pre-change distribution $F$ and the post-change distribution $G'$ differ in expectation by $\delta_{\mu,m}$ and in variance by $\delta_{\xi,m}$ (the paper's Definition 1). [a kind of D8] D19 [data] Royal Mail forecast errors: Forecast errors of the daily number of parcels to be delivered from one Royal Mail delivery office, from 1 July 2020 to 31 December 2020. The underlying data and forecasts are not known to the authors. July to October is the Phase I sample; monitoring starts on 1 November 2020. D23 [variable] date of detection: The date of the observation at which a detector first crosses its threshold. D24 [method] Page's CUSUM detector $D(m,k)$: The sequential detector of a mean change in a series $Y_t$ that follows the change in mean model $$Y_t = \begin{cases} \lambda + \epsilon_t, & t = 1, \ldots, m + k^* \\ \lambda + \epsilon_t + \Delta_m, & t = m + k^* + 1, m + k^* + 2, \ldots \end{cases}$$ with zero-mean variables $\epsilon_t$, a training period of length $m$ without changepoints and a change of size $\Delta_m$ at time $k^*$ after it. With $$\begin{aligned} Q(m,k) &= \sum_{t=m+1}^{m+k} Y_t - \frac{k}{m} \sum_{t=1}^{m} Y_t \\ D(m,k) &= \max_{0 \le i \le k} \left| Q(m,k) - Q(m,i) \right| \end{aligned}$$ the null hypothesis $H_0: \Delta_m = 0$ is rejected in favour of $H_A: \Delta_m \ne 0$ at the stopping time $$\tau_m = \min\{k \ge 1 : D(m,k) \ge \hat{\sigma}_m c_\alpha g(m,k,\gamma)\},$$ where $\hat{\sigma}_m$ is a weakly consistent estimator, from the $m$ training samples, of the constant $\sigma$ of the weak invariance principle, the weight function is $$g(m,k,\gamma) = \sqrt{m}\left(1 + \frac{k}{m}\right)\left(\frac{k}{m+k}\right)^{\gamma}, \quad \gamma \in [0, 1/2).$$ and the critical value $c_\alpha$ is taken from the limit distribution under $H_0$, so that $\lim_{m\to\infty} P(\tau_m < \infty) = \alpha$ under $H_0$ for the chosen false alarm rate $\alpha$ and $\lim_{m\to\infty} P(\tau_m < \infty) = 1$ under $H_A$. All limits are for $m \to \infty$; the model is trained once and not re-trained during monitoring. The paper takes $\gamma = 0$. [source B10] CLAIMS C6 [descriptive; principal; support: empirical_quant E9 (E9.R1, E9.R2) artefacts A5; uses C3, C23; limitations L4; strength low (author); verifiability unverifiable; planning exploratory] On [[D19:the Royal Mail forecast errors]], the [[D2:mean detector]] and the [[D3:squared-error detector]] of the [[D1:forecast error monitoring framework]] both signalled a changepoint on 2 December 2020. scope: setting: one Royal Mail delivery office | geography: United Kingdom | period: 1 July 2020 to 31 December 2020 addresses B1: The closest result found: Aue et al. (2015) apply residual-based monitoring of ARMA models to EEG and IBM data; here only the forecast errors are known, not the model or the data. C1 [formal; principal; extension; support: formal_proof E1; assumes C18, C19; limitations L1; verification unchecked; verifiability open] Under [[D8:the mean change model]], the one-step-ahead forecast errors of a forecasting model that satisfies the two forecasting-model assumptions have a change in mean of size $\Delta_{\mu,m} \ne 0$ at the changepoint: $e_t = b_\mu + \epsilon_{\mu,t}$ for $t \le m + k^*$ and $e_t = b_\mu + \epsilon_{\mu,t} + \Delta_{\mu,m}$ afterwards. scope: data_type: univariate time series | conditions: one-step-ahead forecasts; a single changepoint after the training sample extends B1: Aue et al. (2015) monitor the residuals of ARMA models only; the claim covers the forecast errors of any forecasting model that satisfies the two assumptions. C2 [formal; principal; extension; support: formal_proof E2; assumes C18, C19; verification unchecked; verifiability open] Under [[D9:the mean and variance change model]] and the constant-bias assumption, the change in the mean of the centred squared forecast errors is $$\Delta_{\xi,m} = \mathrm{Var}_{G'}[Y_t|\tilde{y}_t] - \mathrm{Var}_F[Y_t|\tilde{y}_t]$$ when $\delta_{\mu,m} = 0$, and $$\Delta_{\xi,m} = \mathrm{Var}_{G'}[Y_t|\tilde{y}_t] - \mathrm{Var}_F[Y_t|\tilde{y}_t] + \Delta_{\mu,m}^2$$ when $\delta_{\mu,m} \ne 0$ and the incomplete-adjustment assumption holds. scope: data_type: univariate time series | conditions: one-step-ahead forecasts extends B1: Aue et al. (2015) monitor the residuals of ARMA models only; the claim covers the forecast errors of any forecasting model that satisfies the two assumptions. C3 [formal; principal; extension; support: formal_proof E3; assumes C18, C19, C20, C21; uses C2, C24, C25; verification unchecked; verifiability open] For the [[D3:squared-error detector]] under [[D9:the mean and variance change model]], the limit distribution of Theorem 1 of the paper's Supporting Information holds under the null hypothesis of no change, and the limit distribution of its Theorem 2 holds for the stopping time $\tau_{\xi,m}$ under the alternative when the change size $\Delta_{\xi,m}$ is large enough in the sense of that document's Assumptions C to E. scope: conditions: change sizes satisfying Assumptions C to E of the paper's Supporting Information extends B1: Aue et al. (2015) monitor the residuals of ARMA models only; the claim covers the forecast errors of any forecasting model that satisfies the two assumptions. C23 [formal; supporting; extension; support: formal_proof E11; assumes C18, C19, C20, C21; uses C1, C24, C25; verification unchecked; verifiability open] For the [[D2:mean detector]] under [[D8:the mean change model]], the limit distribution of Theorem 1 of the paper's Supporting Information holds under the null hypothesis of no change with $D(m,k) = D_\mu(m,k)$ and $\hat{\sigma}_m = \hat{\sigma}_{\mu,m}$, and the limit distribution of its Theorem 2 holds under the alternative with $\tau_m = \tau_{\mu,m}$ and $c_\alpha = c_{\mu,\alpha}$ when the change size $\Delta_{\mu,m}$ satisfies its Assumptions C to E. scope: data_type: univariate time series | conditions: one-step-ahead forecasts; change sizes satisfying Assumptions C to E of the paper's Supporting Information extends B1: Aue et al. (2015) give the limit distributions for the residuals of ARMA models; the claim gives them for the forecast errors of any forecasting model that satisfies the two assumptions. C18 [formal; supporting; assumption (theorem_condition, unchecked); verifiability not_applicable] The forecasting model has a constant bias before the changepoint: $\mathbb{E}_F[Y_t|\tilde{y}_t] - \mathbb{E}_{\mathcal{F}}[Y_t|\tilde{y}_t] = b_\mu$ for $t = 1, \ldots, m + k^*$, where $b_\mu$ is unknown and may be zero (the paper's Assumption 1, the constant-bias assumption). C19 [formal; supporting; assumption (theorem_condition, unchecked); verifiability not_applicable] After the changepoint the forecasting model does not fully adjust to the change: $$\mathbb{E}_G[Y_t|\tilde{y}_t] - \mathbb{E}_{\mathcal{F}}[Y_t|\tilde{y}_t] = b_\mu + \Delta_{\mu,m}$$ for $t = m + k^* + 1, m + k^* + 2, \ldots$, where $\Delta_{\mu,m} \ne 0$ is a function of the change size in the raw data (the paper's Assumption 2, the incomplete-adjustment assumption). C20 [formal; supporting; assumption (statistical, unchecked); verifiability not_applicable] The zero-mean variables $\epsilon_t$ of the monitored series, here $\epsilon_{\mu,t}$ and $\epsilon_{\xi,t}$ left in the forecast errors after the bias is removed, satisfy the weak invariance principle of Assumptions A and B of the paper's Supporting Information: $\left|\sum_{t=1}^{m} \epsilon_t\right| = O_P(\sqrt{m})$, and there are Wiener processes $\{W_m(i) : i \ge 0, m \ge 1\}$ and a constant $\sigma > 0$ with $$\sup_{1/m \le i < \infty} \frac{1}{(mi)^{1/\nu}} \left| \sum_{t=m+1}^{m+mi} \epsilon_t - \sigma W_m(mi) \right| = O_P(1) \quad \text{for some } \nu > 2$$. C21 [formal; supporting; assumption (theorem_condition, unchecked); verifiability not_applicable] The change is large enough and bounded: the changepoint is $k^* = \lfloor \theta m^\beta \rfloor$ for some $\theta > 0$ and $0 \le \beta < 1$, $\sqrt{m}\,|\Delta_m| \to \infty$ as $m \to \infty$, and $\Delta_m = O(1)$, where $\Delta_m$ is the change size in the monitored series, here $\Delta_{\mu,m}$ or $\Delta_{\xi,m}$, as Assumptions C to E of the paper's Supporting Information require. C24 [formal; supporting; synthesis; support: cited B8; assumes C20; verifiability open] For [[D24:Page's CUSUM detector]] under the null hypothesis of no change, $$\lim_{m\to\infty} P\left(\frac{1}{\hat{\sigma}_m} \sup_{1 \le k < \infty} \frac{D(m,k)}{g(m,k,\gamma)} \le c\right) = P\left(\sup_{0