The claim
On the GRP admissions series, both detectors of the forecast error monitoring framework have a shorter detection delay than model-based sequential changepoint detection, whose minimum possible detection delay on data with yearly seasonality is 24 months.
Scope: setting: A&E admissions in participating hospitals; where: England; data: monthly proportions with yearly seasonality · Argument: W2 · Uses: C13, C14 · Limitations: L5 · Addresses: Taib et al. (2021) · Warrant: W2
Claims it rests on
On the GRP admissions series, the detection delay of the mean detector was 4 data points, counted from the changepoint that the retrospective analysis of Taib et al. (2021) identified.
Scope: setting: A&E admissions in participating hospitals; where: England; period: 2010 to 2020; data: monthly proportions · Data: E10 (R1), artefacts A6 · Limitations: L5 · Used by: C7
On the GRP admissions series, the detection delay of the squared-error detector was 2 time points.
Scope: setting: A&E admissions in participating hospitals; where: England; period: 2010 to 2020; data: monthly proportions · Data: E10 (R2), artefacts A6 · Limitations: L5 · Used by: C7
Evidence
Both detectors applied sequentially to the one-step-ahead forecast errors of a SARMA(1,0,0)(1,0,0) model of the monthly proportion of GRP A&E admissions, fitted on the data up to 2013, with 2013 to 2016 as the Phase I sample of size 36 and monitoring from 2016 (Section 6 and Figure 6 of the paper).
Setup: studied mean detector $D_\mu(m,k)$, squared-error detector $D_\xi(m,k)$ · cases GRP admissions series · procedure forecast error monitoring framework · lens detection delay
Figure: A6 The proportion of GRP A&E admissions, the forecast errors of a SARIMA(1,0,0)(1,0,0)12 model, and the two detectors with their thresholds.
- R1 detection delay = 4 observations (detector: mean detector $D_\mu(m,k)$, reference: the changepoint identified by Taib et al. (2021))
- R2 detection delay = 2 observations (detector: squared-error detector $D_\xi(m,k)$)
A model-based sequential detector must fit the post-change model, here a SARMA(1,0,0)(1,0,0) model with a 12-month season, which the paper states needs at least two years of data after the change; the detectors on the forecast errors signalled four and two observations after it.
Limitations
The application uses one monthly series with a Phase I sample of 36 observations.
Definitions
Methods 5
D1 forecast error monitoring framework: A framework that monitors a forecasting model online by applying sequential changepoint detection to its one-step-ahead forecast errors. For a univariate series and a forecasting model with one-step-ahead point forecast , the forecast errors are The first forecast errors are the Phase I (training) sample; monitoring starts at and stops when a detector, the mean detector or the squared-error detector, crosses its threshold. Any forecasting approach can supply the errors: model-based, machine learning or expert judgement.
D2 · part of D1 mean detector $D_\mu(m,k)$: Page's CUSUM detector applied to the forecast errors of the forecast error monitoring framework: with stopping time where is the standard deviation of the forecast errors in the Phase I sample. The weight function is that of Page's CUSUM detector, and the critical constant comes from the limit distribution of the detector under the null hypothesis. Source: PAGE (1954).
D3 · part of D1 squared-error detector $D_\xi(m,k)$: Page's CUSUM detector applied to the centred squared forecast errors of the forecast error monitoring framework: with stopping time where is the mean of the forecast errors in the Phase I samples and the standard deviation of the centred squared forecast errors in that sample. is that of Page's CUSUM detector and comes from the limit distribution of the detector under the null hypothesis. The paper prints the second sum of without the square, a typographical error that the guarantor, an author of the paper, has confirmed.
D21 model-based sequential changepoint detection: Sequential changepoint detection that fits the model to the data before and after a potential changepoint and decides by a likelihood-ratio style statistic.
D24 Page's CUSUM detector $D(m,k)$: The sequential detector of a mean change in a series that follows the change in mean model with zero-mean variables , a training period of length without changepoints and a change of size at time after it. With the null hypothesis is rejected in favour of at the stopping time where is a weakly consistent estimator, from the training samples, of the constant of the weak invariance principle, the weight function is and the critical value is taken from the limit distribution under , so that under for the chosen false alarm rate and under . All limits are for ; the model is trained once and not re-trained during monitoring. The paper takes . Source: Chu et al. (1996).
Data 1
D20 GRP admissions series: The monthly proportion of A&E admissions related to gallstone-related pathology (GRP) in participating hospitals across England, 2010 to 2020, based on NHS Hospital Episode Statistics. A SARMA(1,0,0)(1,0,0) model fitted to the data up to 2013 gives one-step-ahead forecasts for the rest; 2013 to 2016 is the Phase I sample of size and monitoring starts in 2016. Source: Taib et al. (2021).
Variables and measures 1
D22 detection delay: The number of observations between a change and the time a detector signals it.
Context
- Addresses Taib et al. (2021) · own earlier work · from C7
- Taib et al. (2021) found the changepoint retrospectively, with all the data from 2010 to 2020; the claim is about detecting it sequentially.
References
- Chu, C.-S. J., Stinchcombe, M., White, H., 1996. Monitoring Structural Change. Econometrica 64, 1045. https://doi.org/10.2307/2171955 Used by: D24
- Grundy, T., Killick, R., Svetunkov, I., 2026. Online Detection of Forecast Model Inadequacies Using Forecast Errors. Journal of Time Series Analysis 47, 715–726. https://doi.org/10.1111/jtsa.12843 Used by: A6
- PAGE, E. S., 1954. CONTINUOUS INSPECTION SCHEMES. Biometrika 41, 100–115. https://doi.org/10.1093/biomet/41.1-2.100 Used by: D2
- Taib, A., Killick, R., Hussain, K., Patel, H., Obeidallah, M. R., 2021. Is there seasonal variation in gallstone related admissions in England?. HPB 23, 1732–1743. https://doi.org/10.1016/j.hpb.2021.04.009 Used by: Context, D20
Artefacts
- A6 · figure · data_plot · Grundy et al. (2026) The proportion of GRP A&E admissions, the forecast errors of a SARIMA(1,0,0)(1,0,0)12 model, and the two detectors with their thresholds.
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C7: 39, Weakly supported
Novelty: new, by the authors
- −45 Checkable Verifiability: unverifiable. To remove it: Make the data, code and models the claim rests on available, or state why they cannot be.
- −10 Evidence Rests on an unchecked proof; evidence named without the observations the claim reports; an argument from other claims; cited work only. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): The application uses one monthly series with a Phase I sample of 36 observations. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C13: 54, Weakly supported
- −45 Checkable Verifiability: unverifiable. To remove it: Make the data, code and models the claim rests on available, or state why they cannot be.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): The application uses one monthly series with a Phase I sample of 36 observations. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C14: 54, Weakly supported
- −45 Checkable Verifiability: unverifiable. To remove it: Make the data, code and models the claim rests on available, or state why they cannot be.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): The application uses one monthly series with a Phase I sample of 36 observations. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.