{
 "bundle_of": "C3",
 "schema_version": "0.40",
 "paper": {
  "id": "entrylib:2026.000001",
  "version": 3,
  "title": "Calibration of the prediction intervals of AutoMSARIMA in smooth on M1, M3, Tourism and M4",
  "licence": "CC-BY-4.0",
  "status": "corrected",
  "date": "2026-11-10",
  "status_notice": {
   "date": "2026-11-10",
   "reason_code": "correction",
   "statement": "The test quantities of the claims bound result entries that version 2 had turned into table cells; they now bind the cells, which keep the full precision the verdicts were computed at, and each table names the CSV of the lab results (A10) as its source. The defining equations of D1, D4, D11 and D13 are set as displayed equations.",
   "issued_by": "Ivan Svetunkov (ORCID 0000-0001-7826-0281)"
  }
 },
 "claims": [
  {
   "id": "C3",
   "statement": "On the [[D6:M4 sample]], the [[D11:log-normal ARIMA]] with the orders [[D10:AutoMSARIMA]] selects removes the fall of 80% [[D4:coverage]] over the [[D7:level terciles]] (0.774, 0.783 and 0.752, a spread of 0.031) and lowers [[D13:MCE]] from 0.0386 to 0.0177.",
   "about": {
    "subject": "D11",
    "relation": "removes the fall of coverage over the level terciles and lowers the MCE of",
    "object": [
     "D4",
     "D13"
    ],
    "comparator": "D10",
    "conditions": [
     "D6"
    ]
   },
   "type": "descriptive",
   "role": "principal",
   "novelty": "no_prior_found",
   "support": [
    {
     "kind": "empirical_quant",
     "evidence": "E9",
     "results": [
      "E9.T1.r2",
      "E9.T1.r1"
     ]
    }
   ],
   "provenance": "author_stated",
   "hypothesis": {
    "lab_event": "H9",
    "outcome": "supported"
   },
   "planning": "confirmatory",
   "planned_at": "2026-11-05T18:40:25Z",
   "planned_at_anchor": "git_remote",
   "test": {
    "framework": "deterministic",
    "parameters": {
     "sample": "4500 M4 series as in the plan of H8, same seed"
    },
    "quantities": {
     "sd": {
      "variable": "range of the 80% coverage over the level terciles",
      "conditions": {
       "data": "D6",
       "method": "D11"
      }
     },
     "md": {
      "variable": "D13",
      "conditions": {
       "data": "D6",
       "method": "D11"
      }
     },
     "ma": {
      "variable": "D13",
      "conditions": {
       "data": "D6",
       "method": "D10"
      }
     }
    },
    "supported": "sd <= 0.05 and md <= 0.5*ma",
    "not_supported": "sd > 0.10 or md >= ma",
    "verdict": "computed",
    "runs": 1,
    "lab_plan": "PL13",
    "flags": [
     "code_changed_after_plan"
    ]
   },
   "scope": {
    "data_type": "M4 yearly, quarterly and monthly series",
    "conditions": [
     "backcasting",
     "orders fixed at those AutoMSARIMA selects"
    ]
   },
   "limitations": [
    "L2"
   ],
   "depends_on": [
    "C2"
   ],
   "assessments": [
    {
     "by": "author",
     "strength": "moderate",
     "rationale": "Planned test on series that did not suggest it; the same sample served the test of C8, and the gain in calibration comes with a higher pinball (C14)."
    }
   ],
   "verifiability": "open",
   "verifiability_flags": []
  },
  {
   "id": "C2",
   "statement": "On the [[D6:M4 sample]], the 80% [[D4:coverage]] of [[D10:AutoMSARIMA]] falls with the [[D17:level ratio]]: 0.813, 0.700 and 0.554 from the lowest to the highest of the [[D7:level terciles]], while ETS(MNN) of [[D15:ETS forced and selected]], whose error scale follows the level, covers 0.764, 0.778 and 0.814.",
   "about": {
    "subject": "D10",
    "relation": "has 80% coverage falling with",
    "object": "D4",
    "comparator": "D15",
    "conditions": [
     "D6",
     "D7"
    ]
   },
   "type": "descriptive",
   "role": "principal",
   "novelty": "no_prior_found",
   "support": [
    {
     "kind": "empirical_quant",
     "evidence": "E8",
     "results": [
      "E8.T1.r1",
      "E8.T1.r2"
     ]
    }
   ],
   "provenance": "author_stated",
   "hypothesis": {
    "lab_event": "H8",
    "outcome": "supported"
   },
   "planning": "confirmatory",
   "planned_at": "2026-11-05T18:40:25Z",
   "planned_at_anchor": "git_remote",
   "test": {
    "framework": "deterministic",
    "parameters": {
     "sample": "4500 M4 series (1500 yearly, quarterly, monthly), seed 20261105, terciles of the recent level over the mean level from the first method"
    },
    "quantities": {
     "fa": {
      "variable": "80% coverage in the low tercile minus that in the high tercile",
      "conditions": {
       "data": "D6",
       "method": "D10"
      }
     },
     "fm": {
      "variable": "80% coverage in the low tercile minus that in the high tercile",
      "conditions": {
       "data": "D6",
       "method": "ETS(MNN)"
      }
     }
    },
    "supported": "fa >= 0.10 and fm <= 0.05",
    "not_supported": "fa < 0.05",
    "verdict": "computed",
    "runs": 1,
    "lab_plan": "PL12",
    "flags": [
     "code_changed_after_plan"
    ]
   },
   "depends_on": [
    "C8"
   ],
   "scope": {
    "data_type": "M4 yearly, quarterly and monthly series",
    "conditions": [
     "backcasting",
     "smooth 1.3.0 (Python)"
    ]
   },
   "limitations": [
    "L2",
    "L3"
   ],
   "assumptions": [
    "C25"
   ],
   "assessments": [
    {
     "by": "author",
     "strength": "moderate",
     "rationale": "Confirmed on 4,500 series that did not suggest the hypothesis, with the same pattern on the benchmark (C13); the level ratio is a proxy for the growth of the scale (C25)."
    }
   ],
   "relations": [
    {
     "type": "addresses",
     "reference": "B5",
     "note": "Chatfield discusses why intervals are too narrow in general; this locates the shortfall of one implementation on series whose level has grown."
    }
   ],
   "verifiability": "open",
   "verifiability_flags": []
  },
  {
   "id": "C8",
   "type": "descriptive",
   "role": "supporting",
   "statement": "On the 5,315 [[D5:benchmark]] series, the fall of 80% [[D4:coverage]] over the [[D7:level terciles]] is shared by [[D10:AutoMSARIMA]] (0.786, 0.668, 0.590) and the [[D16:other ARIMA implementations]]: statsforecast AutoARIMA (0.789, 0.667, 0.589), pmdarima (0.770, 0.635, 0.582) and skforecast (0.781, 0.658, 0.585).",
   "about": {
    "subject": "D16",
    "relation": "share the fall of coverage over the level terciles with",
    "object": "D4",
    "comparator": "D10",
    "conditions": [
     "D5"
    ]
   },
   "support": [
    {
     "kind": "empirical_quant",
     "evidence": "E13",
     "results": [
      "E13.T1.r1",
      "E13.T1.r2",
      "E13.T1.r3",
      "E13.T1.r4"
     ]
    }
   ],
   "scope": {
    "data_type": "M1, M3 and Tourism series",
    "conditions": [
     "backcasting for smooth"
    ]
   },
   "novelty": "no_prior_found",
   "assumptions": [
    "C25"
   ],
   "assessments": [
    {
     "by": "author",
     "strength": "moderate",
     "rationale": "Exploratory, on the data that suggested the pattern; confirmed for AutoMSARIMA on M4 (C8)."
    }
   ],
   "planning": "exploratory",
   "provenance": "author_stated",
   "verifiability": "open",
   "verifiability_flags": []
  },
  {
   "id": "C25",
   "type": "descriptive",
   "role": "supporting",
   "statement": "The ratio of the mean of the last third of a series to its overall mean indexes how much the scale of its errors at the forecast origin exceeds the average scale over the sample.",
   "support": [
    {
     "kind": "assumption"
    }
   ],
   "assumption_category": "data",
   "check": "unchecked",
   "provenance": "author_stated",
   "verifiability": "not_applicable",
   "verifiability_flags": []
  }
 ],
 "definitions": [
  {
   "id": "D1",
   "term": "MSARIMA analytic interval",
   "kind": "method",
   "source": "B6",
   "definition": "The quantiles of `predict(interval=\"prediction\")` of a pure ARIMA in smooth 1.3.0, which dispatches to the approximate interval. The ARIMA is the state space model\n$$\\begin{aligned} y_t &= w^\\top v_{t-l} + \\epsilon_t, \\\\ v_t &= F v_{t-l} + g \\epsilon_t, \\\\ \\epsilon_t &\\sim N(0, \\sigma^2). \\end{aligned}$$\nThe quantile at level $\\tau$ and horizon $h$ is\n$$q_{\\tau,h} = \\hat y_{T+h|T} + \\Phi^{-1}(\\tau) \\sqrt{V_h}, \\qquad V_h = \\hat\\sigma^2 \\frac{T}{T-k} c_h,$$\nwith\n$$\\hat\\sigma^2 = \\frac{1}{T} \\sum_{t=1}^{T} e_t^2, \\qquad c_h = 1 + \\sum_{j=1}^{h-1} \\left(w^\\top F^{j-1} g\\right)^2,$$\nwhere $k$ is the number of estimated parameters and $c_h$ the $h$-th diagonal element of the analytic covariance of the state space model, written here for a model with lag 1."
  },
  {
   "id": "D4",
   "term": "coverage",
   "kind": "variable",
   "definition": "For the quantile $q_{\\tau,j}$ at level $\\tau$ and horizon $j$, the share of holdout observations at or below it,\n$$c_\\tau = \\frac{1}{h} \\sum_{j=1}^{h} 1\\left(y_{T+j} \\le q_{\\tau,j}\\right),$$\naveraged first within a series and then over series. cov80 is $c_{0.9} - c_{0.1}$, the coverage of the central 80% interval; q50 is $c_{0.5}$."
  },
  {
   "id": "D5",
   "term": "benchmark",
   "kind": "data",
   "definition": "The 5,315 series of M1 (1,001), M3 (3,003) and Tourism (1,311) from fcompdata 0.1.4, with their own holdouts and seasonal periods; lags $(1, m)$ for seasonal series.",
   "members": [
    "D18",
    "D19",
    "D20"
   ]
  },
  {
   "id": "D6",
   "term": "M4 sample",
   "kind": "data",
   "source": "B4",
   "definition": "1,500 series each of the yearly, quarterly and monthly M4 series, drawn with seed 20261105 without replacement, with the M4 holdouts."
  },
  {
   "id": "D7",
   "term": "level terciles",
   "kind": "design",
   "definition": "Series split into terciles of the [[D17:level ratio]] across the series of a data set; cov80 reported for each tercile, and the spread as the largest minus the smallest of the three."
  },
  {
   "id": "D10",
   "term": "AutoMSARIMA",
   "kind": "method",
   "source": "B6",
   "definition": "`AutoMSARIMA(lags, initial)` in Python and `auto.msarima()` in R of smooth 1.3.0 / 4.6: ARIMA orders up to AR (3,3), I (2,1), MA (3,3) and a constant selected by AICc, normal errors, the [[D1:MSARIMA analytic interval]] for the quantiles. Backcasting initial states unless stated."
  },
  {
   "id": "D11",
   "term": "log-normal ARIMA",
   "kind": "method",
   "source": "B6",
   "definition": "ADAM with `model=\"NNN\"`, an ARIMA and `distribution=\"dlnorm\"`: the ARIMA applies to the logarithms,\n$$\\begin{aligned} \\log y_t &= w^\\top v_{t-l} + \\log(1+\\epsilon_t), \\\\ v_t &= F v_{t-l} + g \\log(1+\\epsilon_t), \\\\ 1+\\epsilon_t &\\sim \\mathrm{logN}(-\\sigma^2/2, \\sigma^2), \\end{aligned}$$\nso the scale of the error is proportional to the level. In the comparisons with fixed orders, the orders and constant are those [[D10:AutoMSARIMA]] selects on the same series."
  },
  {
   "id": "D13",
   "term": "MCE",
   "kind": "variable",
   "definition": "Mean calibration error over the 99 quantile levels $\\tau = 0.01, \\dots, 0.99$:\n$$\\mathrm{MCE} = \\frac{1}{99} \\sum_{\\tau} \\left|\\bar c_\\tau - \\tau\\right|,$$\nwith $\\bar c_\\tau$ the [[D4:coverage]] of $q_\\tau$."
  },
  {
   "id": "D14",
   "term": "scaled pinball",
   "kind": "variable",
   "definition": "Pinball loss averaged over the 99 quantile levels and the horizons, divided by the mean absolute first difference of the in-sample data for real series; in the simulation, by $\\sigma$, and relpin is its ratio to the pinball of the true predictive distribution."
  },
  {
   "id": "D15",
   "term": "ETS forced and selected",
   "kind": "method",
   "source": "B6",
   "definition": "ETS of smooth with backcasting: AutoADAM-style selection among ETS models (`ES(model=\"ZXZ\")`, or ZXN for non-seasonal series), or ETS(ANN) or ETS(MNN) fitted to every series."
  },
  {
   "id": "D16",
   "term": "other ARIMA implementations",
   "kind": "method",
   "definition": "statsforecast AutoARIMA, pmdarima auto_arima and skforecast Arima, as run in the notebook of the smooth benchmark of 2026-11-05, whose saved forecasts were read."
  },
  {
   "id": "D17",
   "term": "level ratio",
   "kind": "variable",
   "definition": "Mean of the last third of the in-sample data of a series divided by the mean of all its in-sample data."
  },
  {
   "id": "D18",
   "term": "M1",
   "kind": "data",
   "source": "B1",
   "definition": "The 1,001 series of the M1 competition with their holdouts."
  },
  {
   "id": "D19",
   "term": "M3",
   "kind": "data",
   "source": "B2",
   "definition": "The 3,003 series of the M3 competition with their holdouts."
  },
  {
   "id": "D20",
   "term": "Tourism",
   "kind": "data",
   "source": "B3",
   "definition": "The 1,311 series of the tourism forecasting competition with their holdouts."
  }
 ],
 "evidence": [
  {
   "id": "E8",
   "profile": "quant",
   "design": "benchmark",
   "description": "[[D10:AutoMSARIMA]] with backcasting and ETS(MNN) fitted to every series of the [[D6:M4 sample]]; cov80 per [[D7:level terciles]] (run RN41, plan PL12).",
   "availability": {
    "level": "public"
   },
   "lab_runs": [
    "RN41"
   ],
   "setup": {
    "studied": [
     "D10",
     "D15"
    ],
    "cases": [
     "D6"
    ],
    "procedure": [
     "D7"
    ],
    "lens": [
     "D4",
     "D13",
     "D17"
    ]
   },
   "execution": {
    "code_version": "40869cb4fed377c6edd2cddc39b66b7ea5820e3a",
    "software": {
     "python": "3.14.4",
     "platform": "Linux-7.0.0-42-generic-x86_64-with-glibc2.43"
    }
   },
   "tables": [
    {
     "id": "T1",
     "description": "AutoMSARIMA and ETS(MNN) on the M4 sample",
     "conditions": {
      "data": "D6"
     },
     "row_labels": [
      "method"
     ],
     "columns": [
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8
       }
      },
      "D13",
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8,
        "D7": "low"
       }
      },
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8,
        "D7": "mid"
       }
      },
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8,
        "D7": "high"
       }
      },
      "80% coverage in the low tercile minus that in the high tercile",
      "share of holdout observations at or below the 0.5 quantile"
     ],
     "rows": [
      {
       "id": "r1",
       "label": [
        "D10"
       ],
       "values": [
        0.6889,
        0.03865,
        0.8128,
        0.6995,
        0.5544,
        0.25849074074074074,
        null
       ]
      },
      {
       "id": "r2",
       "label": [
        "ETS(MNN)"
       ],
       "values": [
        0.7854,
        0.08485,
        0.764,
        0.7779,
        0.8142,
        -0.05018518518518522,
        0.3723
       ]
      }
     ],
     "artefact": "A10"
    }
   ],
   "artefacts": [
    "A10"
   ]
  },
  {
   "id": "E9",
   "profile": "quant",
   "design": "benchmark",
   "description": "[[D10:AutoMSARIMA]] and the [[D11:log-normal ARIMA]] with the same selected orders on the [[D6:M4 sample]]; MCE and cov80 per [[D7:level terciles]] (run RN42, plan PL13).",
   "availability": {
    "level": "public"
   },
   "lab_runs": [
    "RN42"
   ],
   "setup": {
    "studied": [
     "D10",
     "D11"
    ],
    "cases": [
     "D6"
    ],
    "procedure": [
     "D7"
    ],
    "lens": [
     "D4",
     "D13",
     "D14",
     "D17"
    ]
   },
   "execution": {
    "code_version": "40869cb4fed377c6edd2cddc39b66b7ea5820e3a",
    "software": {
     "python": "3.14.4",
     "platform": "Linux-7.0.0-42-generic-x86_64-with-glibc2.43"
    }
   },
   "tables": [
    {
     "id": "T1",
     "description": "AutoMSARIMA and the log-normal ARIMA with the same orders on the M4 sample",
     "conditions": {
      "data": "D6"
     },
     "row_labels": [
      "method"
     ],
     "columns": [
      "D13",
      {
       "variable": "D14",
       "statistic": "median"
      },
      "range of the 80% coverage over the level terciles",
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8
       }
      },
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8,
        "D7": "low"
       }
      },
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8,
        "D7": "mid"
       }
      },
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8,
        "D7": "high"
       }
      }
     ],
     "rows": [
      {
       "id": "r1",
       "label": [
        "D10"
       ],
       "values": [
        0.03864774286070695,
        0.6752,
        0.2585,
        null,
        null,
        null,
        null
       ]
      },
      {
       "id": "r2",
       "label": [
        "D11"
       ],
       "values": [
        0.017709097144281392,
        0.7679,
        0.03108333333333335,
        0.7698,
        0.7742,
        0.7832,
        0.7521
       ]
      }
     ],
     "artefact": "A10"
    }
   ],
   "artefacts": [
    "A10"
   ]
  },
  {
   "id": "E13",
   "profile": "quant",
   "design": "benchmark",
   "description": "Coverage by [[D7:level terciles]] on the 5,315 [[D5:benchmark]] series for [[D10:AutoMSARIMA]] with backcasting (refitted) and for the [[D16:other ARIMA implementations]] (saved forecasts of the smooth benchmark of 2026-11-05); and coverage of ETS and ARIMA on the series where the selected ETS(ANN) of [[D15:ETS forced and selected]] and the selected ARIMA(0,1,1) give the same forecast and width (exploratory run RN15).",
   "availability": {
    "level": "public"
   },
   "lab_runs": [
    "RN15"
   ],
   "setup": {
    "studied": [
     "D10",
     "D16",
     "D15"
    ],
    "cases": [
     "D5"
    ],
    "procedure": [
     "D7"
    ],
    "lens": [
     "D4",
     "D17"
    ]
   },
   "execution": {
    "code_version": "dadd6a147c1e7f36070270bc6e200975b8f73be4",
    "software": {
     "python": "3.14.4",
     "platform": "Linux-7.0.0-42-generic-x86_64-with-glibc2.43"
    }
   },
   "tables": [
    {
     "id": "T1",
     "description": "80% coverage by level tercile of AutoMSARIMA and the other ARIMA implementations on the benchmark",
     "conditions": {
      "data": "D5"
     },
     "row_labels": [
      "method"
     ],
     "columns": [
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8,
        "D7": "low"
       }
      },
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8,
        "D7": "mid"
       }
      },
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8,
        "D7": "high"
       }
      }
     ],
     "rows": [
      {
       "id": "r1",
       "label": [
        "D10"
       ],
       "values": [
        0.7862,
        0.6677,
        0.5897
       ]
      },
      {
       "id": "r2",
       "label": [
        "statsforecast AutoARIMA"
       ],
       "values": [
        0.7891,
        0.6672,
        0.5893
       ]
      },
      {
       "id": "r3",
       "label": [
        "pmdarima auto_arima"
       ],
       "values": [
        0.7702,
        0.6354,
        0.5825
       ]
      },
      {
       "id": "r4",
       "label": [
        "skforecast Arima"
       ],
       "values": [
        0.7809,
        0.6578,
        0.5854
       ]
      }
     ],
     "artefact": "A10"
    },
    {
     "id": "T2",
     "description": "Series of the benchmark where the selected ETS(ANN) and the selected ARIMA(0,1,1) give the same forecast and width",
     "conditions": {
      "data": "D5"
     },
     "row_labels": [
      "method"
     ],
     "columns": [
      "series",
      {
       "variable": "D4",
       "conditions": {
        "nominal level": 0.8
       }
      }
     ],
     "rows": [
      {
       "id": "r1",
       "label": [
        "both"
       ],
       "values": [
        114.0,
        null
       ]
      },
      {
       "id": "r2",
       "label": [
        "ETS"
       ],
       "values": [
        null,
        0.7336
       ]
      },
      {
       "id": "r3",
       "label": [
        "ARIMA"
       ],
       "values": [
        null,
        0.7336
       ]
      }
     ],
     "artefact": "A10"
    }
   ],
   "artefacts": [
    "A10"
   ]
  }
 ],
 "limitations": [
  {
   "id": "L2",
   "category": "other",
   "statement": "The lab record flags plans PL11, PL12 and PL13 with code changed after the plan; the only commit between those plans and their runs updated the claimset skills under .claude/skills, not the analysis scripts.",
   "affects": [
    "C2",
    "C3",
    "C4"
   ],
   "impact": "reduces_precision",
   "mitigation": "none"
  },
  {
   "id": "L3",
   "category": "measurement",
   "statement": "ETS(MNN), the comparator whose coverage stays flat over the level terciles on M4, is badly centred there (q50 of 0.372) and has an MCE of 0.0849, so its flat cov80 does not make it well calibrated.",
   "affects": [
    "C2"
   ],
   "impact": "reduces_transferability",
   "mitigation": "none"
  }
 ],
 "artefacts": [
  {
   "id": "A7",
   "kind": "lab_record",
   "uri": "https://github.com/openforecast-org/smooth/blob/619d23403fa35bfe2f51a4f16effb5d0d3833102/lab/lab.jsonl",
   "version": "sha256:9ec8a2276507e5d7662d670e016119112d54c7ef508f4a71736ecbbf5191a14c",
   "licence": "CC-BY-4.0",
   "availability": {
    "level": "public"
   },
   "description": "Lab record of the study (branch claimset/msarima-interval-calibration): questions, hypotheses, plans, approvals, runs, results and notes as an append-only, hash-chained event log (lab record 0.20, claimset 0.39.0)."
  },
  {
   "id": "A10",
   "kind": "data",
   "uri": "https://entrylib.org/p/2026.000001/v2/files/files/lab-results.csv",
   "licence": "CC-BY-4.0",
   "availability": {
    "level": "public"
   },
   "description": "Every result recorded in the lab record (A7), one row per result: the lab result ID, the run that produced it, the quantity as the analysis script names it, its conditions as JSON and its full-precision value. The tables of the evidence objects hold these values rounded to four significant digits and at least four decimals.",
   "version": "sha256:0e97428c7dc3643e413d3651fe832c8e4696d6ce12177d84bd2a062219f8539b"
  }
 ],
 "references": [
  {
   "id": "B1",
   "target": "10.1002/for.3980010202",
   "id_type": "doi"
  },
  {
   "id": "B2",
   "target": "10.1016/S0169-2070(00)00057-1",
   "id_type": "doi"
  },
  {
   "id": "B3",
   "target": "10.1016/j.ijforecast.2010.04.009",
   "id_type": "doi"
  },
  {
   "id": "B4",
   "target": "10.1016/j.ijforecast.2019.04.014",
   "id_type": "doi"
  },
  {
   "id": "B6",
   "target": "10.1201/9781003452652",
   "id_type": "doi",
   "self": true
  }
 ]
}