The claim
On the Royal Mail forecast errors, the mean detector and the squared-error detector of the forecast error monitoring framework both signalled a changepoint on 2 December 2020.
Scope: setting: one Royal Mail delivery office; where: United Kingdom; period: 1 July 2020 to 31 December 2020 · Data: E9 (R1, R2), artefacts A5 · Uses: C3, C23 · Limitations: L4 · Addresses: B1
Under the mean change model, the one-step-ahead forecast errors of a forecasting model that satisfies the two forecasting-model assumptions have a change in mean of size at the changepoint: for and afterwards.
Scope: data: univariate time series; conditions: one-step-ahead forecasts, a single changepoint after the training sample · Proof: E1 · Assumes: C18, C19 · Limitations: L1 · Extends: B1 · Verification: not checked
Under the mean and variance change model and the constant-bias assumption, the change in the mean of the centred squared forecast errors is when , and when and the incomplete-adjustment assumption holds.
Scope: data: univariate time series; conditions: one-step-ahead forecasts · Proof: E2 · Assumes: C18, C19 · Extends: B1 · Verification: not checked
For the squared-error detector under the mean and variance change model, the limit distribution of Theorem 1 of the paper's Supporting Information holds under the null hypothesis of no change, and the limit distribution of its Theorem 2 holds for the stopping time under the alternative when the change size is large enough in the sense of that document's Assumptions C to E.
Scope: conditions: change sizes satisfying Assumptions C to E of the paper's Supporting Information · Proof: E3 · Uses: C2, C24, C25 · Assumes: C18, C19, C20, C21 · Extends: B1 · Verification: not checked
Claims it rests on
For the mean detector under the mean change model, the limit distribution of Theorem 1 of the paper's Supporting Information holds under the null hypothesis of no change with and , and the limit distribution of its Theorem 2 holds under the alternative with and when the change size satisfies its Assumptions C to E.
Scope: data: univariate time series; conditions: one-step-ahead forecasts, change sizes satisfying Assumptions C to E of the paper's Supporting Information · Proof: E11 · Uses: C1, C24, C25 · Assumes: C18, C19, C20, C21 · Extends: B1 · Verification: not checked · Used by: C6
For Page's CUSUM detector under the null hypothesis of no change, for every real , where is a standard Brownian motion (Theorem 1 of the paper's Supporting Information, from Fremdt 2015).
Scope: conditions: no change in the monitored series, limit for a training sample size going to infinity · Cited: Fremdt (2015) · Assumes: C20 · Used by: C3, C23
For Page's CUSUM detector under the alternative of a mean change, for every real , where is the unique solution of and with equal to when , to when , and to when , where is the standard normal distribution function, a standard Brownian motion and the unique solution of with (Theorem 2 of the paper's Supporting Information, from Fremdt 2014).
Scope: conditions: a change in mean satisfying Assumptions C to E of the paper's Supporting Information, limit for a training sample size going to infinity · Cited: Fremdt (2014) · Assumes: C20, C21 · Used by: C3, C23
Evidence
Derivation of the mean change in the forecast errors from the two forecasting-model assumptions (Section 2.1 of the paper).
Foundation: Kolmogorov probability axioms · Assumes: C18, C19
Proof of Proposition 1 of the paper: the change size of the centred squared forecast errors.
Foundation: Kolmogorov probability axioms · Assumes: C18, C19
Corollary 1 and Proposition 2 of the paper: the limit distributions of the squared-error detector follow from Proposition 1 and from Theorems 1 and 2 of the paper's Supporting Information, the limit theory of Page's CUSUM.
Foundation: Kolmogorov probability axioms · Assumes: C18, C19, C20, C21
- The centred squared forecast errors follow a mean change model with change size and zero-mean errors . (Proposition 1; from C2, C20)
- Under the null hypothesis the limit distribution of Theorem 1 of the Supporting Information holds with and . (Theorem 1 of the Supporting Information; from E3.1, C24)
- Under the alternative, with satisfying Assumptions C to E, the limit distribution of Theorem 2 of the Supporting Information holds with and . (Theorem 2 of the Supporting Information; from E3.1, C21, C25)
Both detectors applied sequentially to the forecast errors of daily parcel volumes of one Royal Mail delivery office, with July to October 2020 as the Phase I sample and monitoring from 1 November 2020, using the theoretical thresholds (Section 5.1 and Figure 5 of the paper).
Setup: studied mean detector $D_\mu(m,k)$, squared-error detector $D_\xi(m,k)$ · cases Royal Mail forecast errors · procedure forecast error monitoring framework · lens date of detection
Figure: A5 Forecast errors of the number of parcels to be delivered from one Royal Mail delivery office in 2020, and the two detectors with their thresholds.
- R1 date of detection = 2020-12-02 (detector: mean detector $D_\mu(m,k)$)
- R2 date of detection = 2020-12-02 (detector: squared-error detector $D_\xi(m,k)$)
Corollaries 1 and 2 of the paper's Supporting Information: the limit distributions of the mean detector follow from the mean change in the forecast errors and from Theorems 1 and 2.
Foundation: Kolmogorov probability axioms · Assumes: C18, C19, C20, C21
- The forecast errors follow the change in mean model of Page's CUSUM detector with , and zero-mean errors . (substitution; from C1, C20)
- Under the null hypothesis the limit distribution of Theorem 1 holds with and . (Theorem 1 of the Supporting Information; from E11.1, C24)
- Under the alternative, with satisfying Assumptions C to E, the limit distribution of Theorem 2 holds with and . (Theorem 2 of the Supporting Information; from E11.1, C21, C25)
Limitations
Forecasting models that adapt to a change in mean, such as ETS models with a positive smoothing parameter, do not satisfy the incomplete-adjustment assumption, so a mean change does not stay in their forecast errors.
The start of the Christmas peak season at the delivery office is inferred from the rise in the mean and variance of the forecast errors; the underlying parcel volumes and forecasts are not known to the authors.
The application uses one monthly series with a Phase I sample of 36 observations.
Assumptions
The forecasting model has a constant bias before the changepoint: for , where is unknown and may be zero (the paper's Assumption 1, the constant-bias assumption). Rested on by C1, C2, C3, C23, E1, E2, E3, E11.
After the changepoint the forecasting model does not fully adjust to the change: for , where is a function of the change size in the raw data (the paper's Assumption 2, the incomplete-adjustment assumption). Rested on by C1, C2, C3, C23, E1, E2, E3, E11.
The zero-mean variables of the monitored series, here and left in the forecast errors after the bias is removed, satisfy the weak invariance principle of Assumptions A and B of the paper's Supporting Information: , and there are Wiener processes and a constant with Rested on by C3, C23, C24, C25, E3, E11.
Definitions
Methods 4
D1 forecast error monitoring framework: A framework that monitors a forecasting model online by applying sequential changepoint detection to its one-step-ahead forecast errors. For a univariate series and a forecasting model with one-step-ahead point forecast , the forecast errors are The first forecast errors are the Phase I (training) sample; monitoring starts at and stops when a detector, the mean detector or the squared-error detector, crosses its threshold. Any forecasting approach can supply the errors: model-based, machine learning or expert judgement.
D2 · part of D1 mean detector $D_\mu(m,k)$: Page's CUSUM detector applied to the forecast errors of the forecast error monitoring framework: with stopping time where is the standard deviation of the forecast errors in the Phase I sample. The weight function is that of Page's CUSUM detector, and the critical constant comes from the limit distribution of the detector under the null hypothesis. Source: PAGE (1954).
D3 · part of D1 squared-error detector $D_\xi(m,k)$: Page's CUSUM detector applied to the centred squared forecast errors of the forecast error monitoring framework: with stopping time where is the mean of the forecast errors in the Phase I samples and the standard deviation of the centred squared forecast errors in that sample. is that of Page's CUSUM detector and comes from the limit distribution of the detector under the null hypothesis. The paper prints the second sum of without the square, a typographical error that the guarantor, an author of the paper, has confirmed.
D24 Page's CUSUM detector $D(m,k)$: The sequential detector of a mean change in a series that follows the change in mean model with zero-mean variables , a training period of length without changepoints and a change of size at time after it. With the null hypothesis is rejected in favour of at the stopping time where is a weakly consistent estimator, from the training samples, of the constant of the weak invariance principle, the weight function is and the critical value is taken from the limit distribution under , so that under for the chosen false alarm rate and under . All limits are for ; the model is trained once and not re-trained during monitoring. The paper takes . Source: Chu et al. (1996).
Data 3
D8 mean change model: A data-generating process with a changepoint at an unknown location : where and differ in expectation by . The raw data may be temporally dependent, trended, seasonal or depend on exogenous variables.
D9 · a kind of D8 mean and variance change model: The mean change model extended so that the pre-change distribution and the post-change distribution differ in expectation by and in variance by (the paper's Definition 1).
D19 Royal Mail forecast errors: Forecast errors of the daily number of parcels to be delivered from one Royal Mail delivery office, from 1 July 2020 to 31 December 2020. The underlying data and forecasts are not known to the authors. July to October is the Phase I sample; monitoring starts on 1 November 2020.
Variables and measures 1
D23 date of detection: The date of the observation at which a detector first crosses its threshold.
Context
- Addresses B1 · from C6
- The closest result found: Aue et al. (2015) apply residual-based monitoring of ARMA models to EEG and IBM data; here only the forecast errors are known, not the model or the data.
- Extends B1 · from C1
- Aue et al. (2015) monitor the residuals of ARMA models only; the claim covers the forecast errors of any forecasting model that satisfies the two assumptions.
- Extends B1 · from C2
- Aue et al. (2015) monitor the residuals of ARMA models only; the claim covers the forecast errors of any forecasting model that satisfies the two assumptions.
- Extends B1 · from C3
- Aue et al. (2015) monitor the residuals of ARMA models only; the claim covers the forecast errors of any forecasting model that satisfies the two assumptions.
- Extends B1 · from C23
- Aue et al. (2015) give the limit distributions for the residuals of ARMA models; the claim gives them for the forecast errors of any forecasting model that satisfies the two assumptions.
References
- Chu, C.-S. J., Stinchcombe, M., White, H., 1996. Monitoring Structural Change. Econometrica 64, 1045. https://doi.org/10.2307/2171955 Used by: D24
- Fremdt, S., 2014. Asymptotic distribution of the delay time in page's sequential procedure. Journal of Statistical Planning and Inference 145, 74–91. https://doi.org/10.1016/j.jspi.2013.09.001 Used by: C25
- Fremdt, S., 2015. Page's sequential procedure for change-point detection in time series regression. Statistics 49, 128–155. https://doi.org/10.1080/02331888.2013.870568 Used by: C24
- Grundy, T., Killick, R., Svetunkov, I., 2026. Online Detection of Forecast Model Inadequacies Using Forecast Errors. Journal of Time Series Analysis 47, 715–726. https://doi.org/10.1111/jtsa.12843 Used by: A5
- PAGE, E. S., 1954. CONTINUOUS INSPECTION SCHEMES. Biometrika 41, 100–115. https://doi.org/10.1093/biomet/41.1-2.100 Used by: D2
Artefacts
- A5 · figure · data_plot · Grundy et al. (2026) Forecast errors of the number of parcels to be delivered from one Royal Mail delivery office in 2020, and the two detectors with their thresholds.
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C6: 15, Weakly supported
Novelty: new, by the authors
- −45 Checkable Verifiability: unverifiable. To remove it: Make the data, code and models the claim rests on available, or state why they cannot be.
- −25 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Assumption C18 is not checked: The forecasting model has a constant bias before the changepoint: for , where is unknown and may be zero (the paper's Assumption 1, the constant-bias assumption). To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −3 Open problems Limitation L4 (reduces precision, not mitigated): The start of the Christmas peak season at the delivery office is inferred from the rise in the mean and variance of the forecast errors; the underlying parcel volumes and forecasts are not known to the authors. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Assumption C19 is not checked: After the changepoint the forecasting model does not fully adjust to the change: for , where is a function of the change size in the raw data (the paper's Assumption 2, the incomplete-adjustment assumption). To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C20 is not checked: The zero-mean variables of the monitored series, here and left in the forecast errors after the bias is removed, satisfy the weak invariance principle of Assumptions A and B of the paper's Supporting Information: , and there are Wiener processes and a constant with . To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C21 is not checked: The change is large enough and bounded: the changepoint is for some and , as , and , where is the change size in the monitored series, here or , as Assumptions C to E of the paper's Supporting Information require. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Limitation L1 (reduces transferability, not mitigated): Forecasting models that adapt to a change in mean, such as ETS models with a positive smoothing parameter, do not satisfy the incomplete-adjustment assumption, so a mean change does not stay in their forecast errors. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C1: 70, Partly supported
Novelty: extends earlier work, by the authors
- −25 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −3 Open problems Assumption C18 is not checked: The forecasting model has a constant bias before the changepoint: for , where is unknown and may be zero (the paper's Assumption 1, the constant-bias assumption). To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C19 is not checked: After the changepoint the forecasting model does not fully adjust to the change: for , where is a function of the change size in the raw data (the paper's Assumption 2, the incomplete-adjustment assumption). To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Limitation L1 (reduces transferability, not mitigated): Forecasting models that adapt to a change in mean, such as ETS models with a positive smoothing parameter, do not satisfy the incomplete-adjustment assumption, so a mean change does not stay in their forecast errors. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C2: 71, Partly supported
Novelty: extends earlier work, by the authors
- −25 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −3 Open problems Assumption C18 is not checked: The forecasting model has a constant bias before the changepoint: for , where is unknown and may be zero (the paper's Assumption 1, the constant-bias assumption). To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C19 is not checked: After the changepoint the forecasting model does not fully adjust to the change: for , where is a function of the change size in the raw data (the paper's Assumption 2, the incomplete-adjustment assumption). To remove it: Test the assumption, or analyse how sensitive the claim is to it.
C3: 69, Partly supported
Novelty: extends earlier work, by the authors
- −25 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −3 Open problems Assumption C18 is not checked: The forecasting model has a constant bias before the changepoint: for , where is unknown and may be zero (the paper's Assumption 1, the constant-bias assumption). To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C19 is not checked: After the changepoint the forecasting model does not fully adjust to the change: for , where is a function of the change size in the raw data (the paper's Assumption 2, the incomplete-adjustment assumption). To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C20 is not checked: The zero-mean variables of the monitored series, here and left in the forecast errors after the bias is removed, satisfy the weak invariance principle of Assumptions A and B of the paper's Supporting Information: , and there are Wiener processes and a constant with . To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C21 is not checked: The change is large enough and bounded: the changepoint is for some and , as , and , where is the change size in the monitored series, here or , as Assumptions C to E of the paper's Supporting Information require. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
C23: 68, Partly supported
Novelty: extends earlier work, by the authors
- −25 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −3 Open problems Assumption C18 is not checked: The forecasting model has a constant bias before the changepoint: for , where is unknown and may be zero (the paper's Assumption 1, the constant-bias assumption). To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C19 is not checked: After the changepoint the forecasting model does not fully adjust to the change: for , where is a function of the change size in the raw data (the paper's Assumption 2, the incomplete-adjustment assumption). To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C20 is not checked: The zero-mean variables of the monitored series, here and left in the forecast errors after the bias is removed, satisfy the weak invariance principle of Assumptions A and B of the paper's Supporting Information: , and there are Wiener processes and a constant with . To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C21 is not checked: The change is large enough and bounded: the changepoint is for some and , as , and , where is the change size in the monitored series, here or , as Assumptions C to E of the paper's Supporting Information require. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Limitation L1 (reduces transferability, not mitigated): Forecasting models that adapt to a change in mean, such as ETS models with a positive smoothing parameter, do not satisfy the incomplete-adjustment assumption, so a mean change does not stay in their forecast errors. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C24: 77, Partly supported
Novelty: summarises earlier work, by the authors
- −20 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −3 Open problems Assumption C20 is not checked: The zero-mean variables of the monitored series, here and left in the forecast errors after the bias is removed, satisfy the weak invariance principle of Assumptions A and B of the paper's Supporting Information: , and there are Wiener processes and a constant with . To remove it: Test the assumption, or analyse how sensitive the claim is to it.
C25: 76, Partly supported
Novelty: summarises earlier work, by the authors
- −20 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −3 Open problems Assumption C20 is not checked: The zero-mean variables of the monitored series, here and left in the forecast errors after the bias is removed, satisfy the weak invariance principle of Assumptions A and B of the paper's Supporting Information: , and there are Wiener processes and a constant with . To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Assumption C21 is not checked: The change is large enough and bounded: the changepoint is for some and , as , and , where is the change size in the monitored series, here or , as Assumptions C to E of the paper's Supporting Information require. To remove it: Test the assumption, or analyse how sensitive the claim is to it.