You are reading a research paper published on EntryLib as separate claims, each with its evidence. How to read this page

updated · Version 2 · 17 November 2026 · Statistics and Probability

This is version 2. The current version is 3.

Updated · 17 November 2026 at 22:21 UTC

Nothing the paper claims has changed. This version adds what a reader finds their way by: a name and a summary in plain language for each of the 17 claims, the symbol of six defined terms, and five relations between the paper's own claims (C13 supports C5, C6, C7 and C11; C15 qualifies C12), each relation between the simulations and the proved claims now naming the table rows it concerns. The guarantor approved the names and summaries on 2026-11-17. The paper's subfield is now Statistics and Probability, chosen by the guarantor: the classifier had assigned Cognitive Neuroscience.

By Ivan Svetunkov (ORCID 0000-0001-7826-0281)

What does sMAPE reward? An analytical and simulation study

Responsible researcher: Ivan Svetunkov (0000-0001-7826-0281)

CC-BY-4.0 · Keywords: forecast evaluation, error measures, sMAPE, point forecasts, consistent scoring functions, proof, simulation study

Contributions
  • Claude Code · derivations and simulations (AI agent): Methodology, Formal analysis, Software, Investigation, Visualization, Writing – original draft
  • Ivan Svetunkov (0000-0001-7826-0281): Conceptualization, Supervision, Validation, Writing – review & editing

Made with claimset-lab.

Text file for AI

Text file: the whole paper as plain text, to attach when your assistant cannot open the address.

The message

Read https://entrylib.org/p/[email protected], a research paper written as claims with their evidence. Give me a brief summary of its main findings and how well each is supported. If you cannot open the link, tell me instead of guessing. Then wait for my questions.

Summary

Which forecast does sMAPE reward? sMAPE has been criticised for its asymmetry and its behaviour near zero, but the forecast that minimises it had been found only in special cases. We show that for positive values sMAPE depends only on the ratio of actual to forecast, not on their difference: it is the hyperbolic tangent of half the absolute log error. So it scores a forecast k times too high and one k times too low the same, and is asymmetric only for equal absolute errors. Being bounded, it has a finite expected value for every distribution. Its target is a median of the actuals, each weighted by how close it is to the forecast by ratio. This is the median when the log of the actual is symmetric and unimodal; otherwise it can move to the side of the median that carries more weight. With two well-separated groups of actuals, the median becomes a local maximum. On intermittent demand the target is 0 once the share of zeros passes a threshold below one half. Otherwise sMAPE aims at a weighted median of the non-zero actuals. As the actuals spread out, expected sMAPE approaches its bound for every forecast, so forecasts become hard to tell apart, and the target moves from the median to the mode of the log. Where sMAPE and MAE both target the median, a test set's sMAPE minimiser estimates it less precisely than the sample median, increasingly so as the actuals spread out.

Highlights

  • SupportS 100For y>0y > 0 and f>0f > 0, the sMAPE term s(y,f)=|y−f|/(|y|+|f|)s(y,f) = |y-f|/(|y|+|f|) depends on yy and ff only through their ratio r=y/fr = y/f: s=|r−1|/(r+1)s = |r - 1|/(r + 1). C1
  • SupportS 100The forecast with the lowest expected sMAPE is the median mm of the actuals in each of two cases. (a) ln⁡y\ln y has a density symmetric about ln⁡m\ln m and unimodal. (b) More generally, yy is positive, takes no single value with positive probability, and its expected sMAPE has a single local minimum; with the forecast at mm, the actuals above and below mm carry equal total weight y/(y+f)2y/(y+f)^2; and 2g(ln⁡m)>E[sech2⁡(ln⁡y−ln⁡m2)tanh⁡(|ln⁡y−ln⁡m|2)],2g(\ln m) > \mathrm E\Big[\operatorname{sech}^2\Big(\frac{\ln y - \ln m}{2}\Big)\tanh\Big(\frac{|\ln y - \ln m|}{2}\Big)\Big], where gg is the density of ln⁡y\ln y, continuous at ln⁡m\ln m. In both cases the sMAPE target is the median. C12
  • SupportS 100Let the actual yy be positive and take no single value with positive probability. A forecast ff has the lowest expected sMAPE only if the actuals below ff and those above it carry equal total weight w(y,f)=y/(y+f)2w(y,f) = y/(y+f)^2: E[w(y,f)𝟏{y<f}]=E[w(y,f)𝟏{y>f}].\mathrm E[w(y,f)\,\mathbf 1\{y<f\}] = \mathrm E[w(y,f)\,\mathbf 1\{y>f\}]. So the sMAPE target is a median of the actuals weighted by ww. An actual's weight is largest when it equals the forecast, is the same for y/f=ry/f = r and 1/r1/r, and falls as y/fy/f moves away from 1 in either direction. If expected sMAPE has a single local minimum, the target lies above the median mm when, with the forecast at mm, the actuals above mm carry more weight than those below it; below the median when those below carry more; and when the two weights are equal, the median is a point where expected sMAPE is flat. The median is then the target if, in addition, 2g(ln⁡m)>E[sech2⁡(ln⁡y−ln⁡m2)tanh⁡(|ln⁡y−ln⁡m|2)],2g(\ln m) > \mathrm E\Big[\operatorname{sech}^2\Big(\frac{\ln y - \ln m}{2}\Big)\tanh\Big(\frac{|\ln y - \ln m|}{2}\Big)\Big], where gg is the density of ln⁡y\ln y, continuous at ln⁡m\ln m; this makes the median a strict local minimum. Without it, the median can be a flat point of inflection, with the target elsewhere. C4
  • SupportS 89Where sMAPE and MAE both target the median (log-symmetric outcomes), the sMAPE minimiser of a test set varies more than the sample median for every spread and test-set size studied: its relative efficiency is below 1 and falls as the actuals spread out, at n=10000n = 10000 from 0.97 to 0.03 on lognormal actuals as σ\sigma grows from 0.25 to 6, and from 1.00 to 0.34 on log-Laplace actuals as bb grows from 0.25 to 6, close to the large-sample values. C15
  • SupportS 95Let ln⁡y\ln y have two modes, at ln⁡m−a\ln m - a and ln⁡m+a\ln m + a (a>0a > 0), with the actuals concentrated near them: a mixture of a distribution near ln⁡m−a\ln m - a and its mirror image about ln⁡m\ln m, with weights pp and 1−p1 - p. (a) With equal weights the median mm lies in the gap between the two groups; when the groups are narrow enough it has a higher expected sMAPE than forecasts near either mode, and it is a local maximum of expected sMAPE; there is a minimiser on each side of the median, and these approach me−am\,e^{-a} and meam\,e^{a} as the groups narrow. (b) With unequal weights and narrow groups, the best forecast is near the mode of the larger group. C6

Context

Extends Tofallis (2015) · from C1
Tofallis (2015) lists the sMAPE term, |y-f|/[(|y|+|f|)/2], among Tornqvist et al.'s measures of relative change; C1 shows it depends on the actual and the forecast only through their ratio.
Qualifies Svetunkov (2026a), C33 · from C2
C33 says sMAPE penalises a forecast below the outcome more than one above it. That holds for two forecasts the same absolute distance away (C2 b); a forecast k times too low and one k times too high score the same (C2 a).
Refines Goodwin and Lawton (1999) · from C2
Goodwin and Lawton (1999) showed that sMAPE is asymmetric: for the same absolute error it scores a forecast above the actual better than one below. C2 proves this (b) and shows that by ratio sMAPE is symmetric (a).
Refines Hyndman and Koehler (2006) · from C2
Hyndman and Koehler (2006) note that for the same actual sMAPE penalises low forecasts more heavily than high ones; C2 shows this holds for equal absolute errors and not for equal ratios.
Refines Hewamalage et al. (2023) · from C2
Hewamalage et al. (2023) note that sMAPE penalises underestimates more than overestimates for the same actual; C2 shows this holds for equal absolute errors and not for equal ratios.
Refines Chen et al. (2017) · from C2
Chen et al. (2017) show a heavier penalty on a forecast 10% under the actual than on one 10% over; C2 shows that forecasts the same factor too high and too low score the same.
Extends Tofallis (2015) · from C3
Tofallis (2015) relates sMAPE to the log accuracy ratio among relative measures; C3 shows the sMAPE term is exactly the hyperbolic tangent of half the absolute log ratio.
Explains Svetunkov (2026b), C5 · from C4
C5 found sMAPE's preferred candidate close to the median on other distributions; C4 shows why: the target is a weighted median, on the side of the median that carries more weight.
Addresses Tofallis (2015) · from C4
Tofallis (2015) states that minimising sMAPE does not produce an established measure of location; C4 identifies the target as a median weighted by y/(y+f)^2.
Qualifies Hewamalage et al. (2023) · from C4
Hewamalage et al. (2023) say sMAPE may tend towards selecting a slightly overestimating model; C4 shows the target lies on the side of the median that carries more weight, above or below it.
Replicates Anderson (1955) · from C5
C5 is an elementary case of Anderson's theorem on symmetric unimodal densities, proved here directly for the expected sMAPE.
Refines Svetunkov (2026b), C6 · from C7
C6 found sMAPE preferring zero on counts with more than half zeros; C7 gives the exact threshold, which is below one half whenever the positive outcomes vary.
Extends Kolassa (2026) · from C7
Kolassa (2026) shows that sMAPE elicits a zero forecast for sufficiently intermittent actuals; C7 gives the share of zeros at which it does, which is below one half.
Extends Hewamalage et al. (2023) · from C9
Hewamalage et al. (2023) note that sMAPE is bounded by 200; C9 adds that its expected value is therefore finite for every distribution and that a best forecast exists.
Supports Svetunkov (2026a), C6 · from C10
C6 recommends against sMAPE for point forecasts; C10 shows that on widely spread actuals every forecast scores close to the worst value and the target moves to the mode of ln y.
Supports Kolassa (2026) · from C10
Kolassa (2026) concludes that sMAPE should not be used; C10 shows that on widely spread actuals every forecast scores close to the worst value.
Explains Svetunkov (2026b), C5 · from C12
C5 found by simulation that sMAPE prefers the median on lognormal and log-Laplace outcomes; C12 proves it for every symmetric unimodal ln y.
Supports Svetunkov (2026b), C11 · from C15
C11 found sMAPE's choices in short test sets spreading more than MAE's; C15 measures the loss of precision for test sets of 6 to 10000 outcomes.
Supports Svetunkov (2026a), C6 · from C15
C6 recommends against sMAPE for point forecasts; C15 shows that even where it targets the median it estimates it less precisely than the sample median.
Supports Kolassa (2026) · from C15
Kolassa (2026) concludes that sMAPE should not be used; C15 shows that even where it targets the median it estimates it less precisely than the sample median.
Extends Hewamalage et al. (2021) · from C17
Hewamalage, Bergmeir and Bandara (2021) evaluate forecasts with msMAPE, Suilin's modification of sMAPE for series with values near zero; C17 shows that it keeps sMAPE's behaviour away from zero and becomes an absolute error near zero.

Definitions

Methods 4

D16 selection design: For each outcome distribution: derive the candidate forecasts and their expected scores; then, for each test-set size nn, draw test sets of nn independent outcomes, score every candidate by every measure on each test set, and count the candidate with the lowest score as the measure's choice (a tie of jj candidates counts 1/j1/j each; a test set in which every candidate scores infinite chooses none).

D17 efficiency design: For each outcome distribution and test-set size nn: draw test sets of nn independent outcomes; in each, compute the sample median and the exact sMAPE minimiser (an outcome of the test set, since the test set's sMAPE is concave in ln⁡f\ln f between consecutive outcomes); over test sets, compare the variances of their logarithms.

D18 flatness design: For families whose spread σ\sigma of ln⁡y\ln y grows from 0.1 to 10: derive the sMAPE target, the expected sMAPE of forecasts at fixed quantile levels and the large-spread approximation of C10; on test sets, count how often each measure scores the target lower than a forecast away from it, and where the test set's sMAPE minimiser falls.

D31 formula checks: Numerical checks of the formulas of the formal claims (claimset-code/smape-target/check_formulas.py, seed 6; check_jump.py for C11): identities and inequalities on 200000 random pairs, derivatives against finite differences, minimisers, balances and thresholds against quadrature and bisection on every setting of the selection design, and the large-spread expansion against its bound.

Variables and measures 13

D1 sMAPE term: The contribution of one actual yy and one point forecast ff to sMAPE: s(y,f)=|y−f|/(|y|+|f|)s(y,f) = |y-f|/(|y|+|f|), between 0 and 1, with s=0s = 0 when y=fy = f (including y=f=0y = f = 0, where the fraction is 0/00/0). sMAPE is 200 times its mean.

D2 sMAPE: Symmetric mean absolute percentage error of a point forecast ff over a test set of actuals y1,…,yny_1, \dots, y_n: 200n∑ts(yt,f)\frac{200}{n}\sum_t s(y_t, f), 200 times the mean of the sMAPE terms. Introduced as the symmetric MAPE by Makridakis (1993), whose denominator had no absolute values; for positive actuals and forecasts the two agree. Source: Makridakis (1993).

D3 expected sMAPE: The expected sMAPE of a fixed forecast ff under the distribution of the actual yy: 200E[s(y,f)]200\,\mathrm{E}[s(y, f)], where E\mathrm E is the expectation over the random actual yy. Computed from the distribution (by integration or exact sums), never from a sample.

D4 log error: For a positive actual yy and a positive forecast ff, z=ln⁡y−ln⁡fz = \ln y - \ln f.

D5 weight: For a positive actual yy and a positive forecast ff, w(y,f)=y/(y+f)2w(y,f) = y/(y+f)^2: how much the actual pulls the forecast that minimises expected sMAPE, since the derivative of E[s(y,f)]\mathrm E[s(y,f)] in ff is 2E[w(y,f)(𝟏{y<f}−𝟏{y>f})]2\,\mathrm E[w(y,f)(\mathbf 1\{y<f\} - \mathbf 1\{y>f\})]. Largest at y=fy = f, the same for y/f=ry/f = r and 1/r1/r.

D8 sMAPE minimiser of a test set: The forecast that minimises the sMAPE of one test set, found numerically (the sample counterpart of the sMAPE target).

D9 relative efficiency: For a test-set size nn and an outcome distribution where sMAPE and MAE both target the median: RE(n)=Var⁡(ln⁡m^n)/Var⁡(ln⁡f^n)\mathrm{RE}(n) = \operatorname{Var}(\ln \hat m_n) / \operatorname{Var}(\ln \hat f_n) over test sets, where m^n\hat m_n is the sample median (the MAE minimiser) and f^n\hat f_n the sMAPE minimiser of the same test set. Above 1, sMAPE's minimiser varies less; below 1, MAE's does.

D10 RMSE: Root mean squared error of a point forecast ff over a test set of actuals y1,…,yny_1, \dots, y_n: 1n∑t(yt−f)2\sqrt{\frac1n\sum_t (y_t-f)^2}.

D11 MAE: Mean absolute error of a point forecast ff over a test set of actuals y1,…,yny_1, \dots, y_n: 1n∑t|yt−f|\frac1n\sum_t |y_t-f|.

D12 MAPE: Mean absolute percentage error of a point forecast ff over a test set of actuals y1,…,yny_1, \dots, y_n: 100n∑t|yt−f|/|yt|\frac{100}{n}\sum_t |y_t-f|/|y_t|; infinite when an actual is zero.

D19 log-MAE: 100 times the mean absolute error of ln⁡f\ln f as a forecast of ln⁡y\ln y: 100n∑t|ln⁡yt−ln⁡f|\frac{100}{n}\sum_t |\ln y_t - \ln f|. Unbounded; sMAPE's small-error limit (C3); minimised by the median.

D20 ranking agreement: For one distribution, one measure, a test-set size nn, the measure's target and a forecast at another quantile level: the probability that a test set of nn outcomes gives the target the strictly lower score (ties count as disagreement); 0.5 is a coin toss.

D28 msMAPE term: The contribution of one actual yy and one forecast ff to the modified sMAPE: s~(y,f)=|y−f|max⁡(|y|+|f|+δ,0.5+δ),δ=0.1;\tilde s(y,f) = \frac{|y-f|}{\max(|y|+|f|+\delta,\ 0.5+\delta)}, \qquad \delta = 0.1; msMAPE is 200 times its mean over a test set. It replaces the denominator |y|+|f||y|+|f| of the sMAPE term (proposed by Suilin, 2017, in code for a Kaggle competition; defined in print by Hewamalage, Bergmeir and Bandara, 2021, Eq. 15, who write the constant δ\delta as ϵ\epsilon and the factor 2 into the measure differently; a constant factor changes no property here). Source: Hewamalage et al. (2021).

Populations 8

D21 inverse Gaussian outcomes: Inverse Gaussian outcomes with mean 100 and shape 1000, 200, 100, 50, 20 or 10 (coefficient of variation 0.32 to 3.16): positive, skewed right, every moment and E[1/y]\mathrm E[1/y] finite.

D22 log-skewed outcomes: y=100exy = 100\,e^x with x=±σ(G−α)/αx = \pm\sigma(G-\alpha)/\sqrt\alpha, G∼Gamma(α,1)G \sim \mathrm{Gamma}(\alpha, 1): ln⁡y\ln y with standard deviation σ\sigma (0.5 or 1; sd in the settings' names) and skewness ±2/α\pm 2/\sqrt\alpha (-2, -1, -0.5, +0.5, +1, +2), and the lognormal of the same σ\sigma (skewness 0). Some have infinite moments: E[y]\mathrm E[y] infinite (σ=1\sigma = 1, skewness +2), E[y2]\mathrm E[y^2] infinite (σ=0.5\sigma = 0.5, skewness +2; σ=1\sigma = 1, skewness +1), E[1/y]\mathrm E[1/y] infinite (σ=1\sigma = 1, skewness -2).

D23 bimodal outcomes: y=100exy = 100\,e^x, xx normal with mean −1-1 (probability pp) or +1+1, standard deviation 0.25; p=0.5p = 0.5 (equal modes), 0.65 (left mode higher), 0.35 (right mode higher).

D24 outcomes straddling zero: Normal outcomes with standard deviation 1 and mean 0, 0.5 or 1: 50%, 31% and 16% negative; E[1/|y|]\mathrm E[1/|y|] infinite.

D25 zero-inflated outcomes: Lognormal outcomes (median 100, σ\sigma 0.5 or 1) replaced by 0 with probability p0p_0: 0.25, 0.42, 0.47, 0.6 for σ=0.5\sigma = 0.5 and 0.25, 0.37, 0.42, 0.6 for σ=1\sigma = 1, on both sides of the zero threshold of C7 (0.4469 and 0.3940).

D26 log-symmetric outcomes: Lognormal outcomes with σ\sigma, and log-Laplace outcomes with scale bb, each 0.25, 0.5, 1, 1.5, 2, 3, 4 or 6, median 100: ln⁡y\ln y symmetric and strictly unimodal, so that sMAPE and MAE both target the median (C5).

D27 widening outcomes: ln⁡y=ln⁡100+σε\ln y = \ln 100 + \sigma\varepsilon with ε\varepsilon standard normal (lognormal), standard Gumbel (log-Gumbel right, ln⁡y\ln y skewed right) or minus standard Gumbel (log-Gumbel left), for σ\sigma = 0.1, 0.25, 0.5, 1, 1.5, 2, 3, 4, 6 and 10.

D30 Poisson counts: Poisson outcomes with means 0.3, 0.5, 1, 1.5, 3 and 10: counts whose share of zeros falls from 0.74 to almost 0.

Concepts 6

D6 sMAPE target: The value or values of ff that minimise expected sMAPE for a given distribution of the actual yy, arg⁡minf⁡E[s(y,f)]\arg\min_f \mathrm E[s(y,f)]: the functional sMAPE rewards. It may be a single value or several.

D7 quantile profile: For an outcome distribution with quantile function QQ and a forecast ff, the pull of the outcome at each quantile level uu, u↦w(Q(u),f)u \mapsto w(Q(u), f) on (0,1)(0,1); the minimiser's quantile level balances its integrals below and above.

D13 candidate forecast: A point forecast fixed from the outcome distribution before any test set is drawn and the same for every test set: a quantile at a level from 0.01 to 0.99, the mean, the forecast 0, the log-mode, the minimiser of expected MAPE, or a local minimiser of expected sMAPE. It is never estimated from a sample.

D14 preferred candidate: Of the candidate forecasts for one distribution, the one a measure gives the lowest expected score, or, where every expected score is infinite but their differences are finite, the one the differences rank first (C8); none where no forecast is ranked first.

D15 derived value: A value computed from the outcome distribution itself by numerical integration of its density (Gauss-Legendre quadrature and a convolution on a grid of 524288 points of ln⁡y\ln y), never from a sample: the expected scores, the local minima of expected sMAPE, the formulas of the hypotheses.

D29 msMAPE target: The value or values of ff that minimise the expected msMAPE term for a given distribution of the actual: the functional msMAPE rewards, as the sMAPE target is the one sMAPE rewards.

Principal claims

SupportS 100C1 · formal · planned test · novelty: extends earlier work, by the authors

For y>0y > 0 and f>0f > 0, the sMAPE term s(y,f)=|y−f|/(|y|+|f|)s(y,f) = |y-f|/(|y|+|f|) depends on yy and ff only through their ratio r=y/fr = y/f: s=|r−1|/(r+1)s = |r - 1|/(r + 1).

SupportS 100C3 · formal · planned test · novelty: extends earlier work, by the authors

For y>0y > 0 and f>0f > 0, the sMAPE term is s=tanh⁡(|ln⁡y−ln⁡f|/2)s = \tanh(|\ln y - \ln f|/2) (log error ln⁡y−ln⁡f\ln y - \ln f). (a) It is 0 when y=fy = f, strictly increasing and concave in |ln⁡y−ln⁡f||\ln y - \ln f|, and bounded by 1, which it approaches as |ln⁡y−ln⁡f||\ln y - \ln f| grows. (b) 0≤12|ln⁡y−ln⁡f|−s≤124|ln⁡y−ln⁡f|30 \le \tfrac12|\ln y - \ln f| - s \le \tfrac1{24}|\ln y - \ln f|^3; so when all errors are small, sMAPE is about 100 times the mean absolute error of ln⁡f\ln f as a forecast of ln⁡y\ln y, and like it is minimised by the median.

SupportS 100C4 · formal · planned test · novelty: new, by the authors

Let the actual yy be positive and take no single value with positive probability. A forecast ff has the lowest expected sMAPE only if the actuals below ff and those above it carry equal total weight w(y,f)=y/(y+f)2w(y,f) = y/(y+f)^2: E[w(y,f)𝟏{y<f}]=E[w(y,f)𝟏{y>f}].\mathrm E[w(y,f)\,\mathbf 1\{y<f\}] = \mathrm E[w(y,f)\,\mathbf 1\{y>f\}]. So the sMAPE target is a median of the actuals weighted by ww. An actual's weight is largest when it equals the forecast, is the same for y/f=ry/f = r and 1/r1/r, and falls as y/fy/f moves away from 1 in either direction. If expected sMAPE has a single local minimum, the target lies above the median mm when, with the forecast at mm, the actuals above mm carry more weight than those below it; below the median when those below carry more; and when the two weights are equal, the median is a point where expected sMAPE is flat. The median is then the target if, in addition, 2g(ln⁡m)>E[sech2⁡(ln⁡y−ln⁡m2)tanh⁡(|ln⁡y−ln⁡m|2)],2g(\ln m) > \mathrm E\Big[\operatorname{sech}^2\Big(\frac{\ln y - \ln m}{2}\Big)\tanh\Big(\frac{|\ln y - \ln m|}{2}\Big)\Big], where gg is the density of ln⁡y\ln y, continuous at ln⁡m\ln m; this makes the median a strict local minimum. Without it, the median can be a flat point of inflection, with the target elsewhere.

SupportS 95C6 · formal · planned test · novelty: new, by the authors

Let ln⁡y\ln y have two modes, at ln⁡m−a\ln m - a and ln⁡m+a\ln m + a (a>0a > 0), with the actuals concentrated near them: a mixture of a distribution near ln⁡m−a\ln m - a and its mirror image about ln⁡m\ln m, with weights pp and 1−p1 - p. (a) With equal weights the median mm lies in the gap between the two groups; when the groups are narrow enough it has a higher expected sMAPE than forecasts near either mode, and it is a local maximum of expected sMAPE; there is a minimiser on each side of the median, and these approach me−am\,e^{-a} and meam\,e^{a} as the groups narrow. (b) With unequal weights and narrow groups, the best forecast is near the mode of the larger group.

SupportS 100C7 · formal · planned test · novelty: extends earlier work, by the authors

A forecast of the opposite sign to the actual, a zero forecast of a non-zero actual, and a non-zero forecast of a zero actual all give the sMAPE term its largest value, 1. Let the actual yy be 0 with probability p0p_0 and otherwise positive, and s+s_+ the lowest expected ss a positive forecast reaches on the positive actuals. The forecast 0 is the sMAPE target when p0>(1−s+)/(2−s+)p_0 > (1 - s_+)/(2 - s_+), and a positive forecast does when the inequality is reversed; the threshold is below 1/21/2 whenever s+>0s_+ > 0.

SupportS 100C9 · formal · planned test · novelty: extends earlier work, by the authors

For actuals and forecasts of any sign, 0≤s≤10 \le s \le 1. So the expected sMAPE of every forecast is finite, between 0 and 200, whatever the distribution of the actuals and without any moment of it being finite (unlike RMSE, MAE and MAPE, C8). For a positive random actual yy, at least one forecast has the lowest expected sMAPE.

SupportS 100C10 · formal · planned test · novelty: new, by the authors

Let ln⁡y=μ+σε\ln y = \mu + \sigma\varepsilon, where ε\varepsilon has a fixed density hh with bounded continuous first and second derivatives, and write a forecast as f=eμ+σcf = e^{\mu + \sigma c} (its quantile level, the probability that ε≤c\varepsilon \le c, does not depend on σ\sigma). As σ\sigma grows, E[s(y,f)]=1−4ln⁡2h(c)/σ+O(σ−3)\mathrm E[s(y,f)] = 1 - 4\ln 2\,h(c)/\sigma + O(\sigma^{-3}) uniformly in cc. (a) The expected sMAPE of every forecast tends to its bound 200, and the difference between forecasts at two fixed quantile levels shrinks like 1/σ1/\sigma. (b) When hh has a single highest point c0c_0, the quantile level of the sMAPE target tends to that of c0c_0, the mode of ln⁡y\ln y, which differs from 1/2 when the mode of ε\varepsilon is not its median. (c) As σ→0\sigma \to 0, when ε\varepsilon has a unique median and a finite E|ε|3\mathrm E|\varepsilon|^3, the quantile level of the target tends to 1/2: sMAPE targets the median for narrow distributions and the mode of ln⁡y\ln y for wide ones.

SupportS 100C12 · formal · planned test · novelty: new, by the authors

The forecast with the lowest expected sMAPE is the median mm of the actuals in each of two cases. (a) ln⁡y\ln y has a density symmetric about ln⁡m\ln m and unimodal. (b) More generally, yy is positive, takes no single value with positive probability, and its expected sMAPE has a single local minimum; with the forecast at mm, the actuals above and below mm carry equal total weight y/(y+f)2y/(y+f)^2; and 2g(ln⁡m)>E[sech2⁡(ln⁡y−ln⁡m2)tanh⁡(|ln⁡y−ln⁡m|2)],2g(\ln m) > \mathrm E\Big[\operatorname{sech}^2\Big(\frac{\ln y - \ln m}{2}\Big)\tanh\Big(\frac{|\ln y - \ln m|}{2}\Big)\Big], where gg is the density of ln⁡y\ln y, continuous at ln⁡m\ln m. In both cases the sMAPE target is the median.

SupportS 89C15 · descriptive · planned estimate · novelty: new, by the authors

Where sMAPE and MAE both target the median (log-symmetric outcomes), the sMAPE minimiser of a test set varies more than the sample median for every spread and test-set size studied: its relative efficiency is below 1 and falls as the actuals spread out, at n=10000n = 10000 from 0.97 to 0.03 on lognormal actuals as σ\sigma grows from 0.25 to 6, and from 1.00 to 0.34 on log-Laplace actuals as bb grows from 0.25 to 6, close to the large-sample values.

SupportS 100C17 · formal · planned test · novelty: extends earlier work, by the authors

The msMAPE term has the properties of the sMAPE term except near zero. (a) Where |y|+|f|≥0.5|y|+|f| \ge 0.5, s~(y,f)=s(y,f)|y|+|f||y|+|f|+δ:\tilde s(y,f) = s(y,f)\,\frac{|y|+|f|}{|y|+|f|+\delta}: the sMAPE term shrunk by a factor between 5/65/6 and 1 that tends to 1 as |y|+|f||y|+|f| grows. So for actuals and forecasts large compared with δ\delta, s~\tilde s is close to ss, though no longer exactly a function of the ratio y/fy/f; on counts, where every actual is 0 or at least 1, this case covers every pair except a zero actual with |f|<0.5|f| < 0.5. (b) Where |y|+|f|<0.5|y|+|f| < 0.5, s~(y,f)=|y−f|/(0.5+δ)\tilde s(y,f) = |y-f|/(0.5+\delta), an absolute error: a zero forecast of a zero actual scores 0, and a forecast ff with |f|<0.5|f| < 0.5 of a zero actual scores |f|/(0.5+δ)|f|/(0.5+\delta), where the sMAPE term gives every non-zero forecast of a zero actual its largest value, 1. (c) 0≤s~<10 \le \tilde s < 1 everywhere, so the expected msMAPE of every forecast is finite.

Supporting claims

SupportS 100C2 · formal · planned test · novelty: new, by the authors

For y>0y > 0 and f>0f > 0: (a) replacing r=y/fr = y/f by 1/r1/r leaves the sMAPE term unchanged, so a forecast kk times too high (f=kyf = ky) and one kk times too low (f=y/kf = y/k) score the same; (b) of two forecasts the same absolute distance dd from yy (0<d<y0 < d < y), the higher one scores better: s(y,y+d)=d2y+d<d2y−d=s(y,y−d).s(y, y+d) = \frac{d}{2y+d} < \frac{d}{2y-d} = s(y, y-d).

SupportS 100C5 · formal · planned test · novelty: replicates earlier work, by the authors

Let ln⁡y\ln y have a density gg symmetric about ln⁡m\ln m, g(ln⁡m+v)=g(ln⁡m−v)g(\ln m+v) = g(\ln m-v), and unimodal: g(ln⁡m+v1)≥g(ln⁡m+v2)g(\ln m+v_1) \ge g(\ln m+v_2) whenever |v1|≤|v2||v_1| \le |v_2|, strictly on a set of positive probability. Then the median mm of yy is the only sMAPE target.

SupportS 100C8 · formal · planned test · novelty: new, by the authors

For a positive random actual yy: (a) RMSE: E[(y−f)2]\mathrm E[(y-f)^2] is finite exactly when E[y2]\mathrm E[y^2] is; when E[y2]\mathrm E[y^2] is infinite but E[y]\mathrm E[y] finite, every difference between two forecasts f1f_1 and f2f_2, E[(y−f1)2−(y−f2)2]=(f1−f2)(f1+f2−2E[y]),\mathrm E[(y-f_1)^2 - (y-f_2)^2] = (f_1-f_2)(f_1+f_2-2\,\mathrm E[y]), is finite and the mean is preferred to every other forecast; when E[y]\mathrm E[y] is infinite no forecast is preferred to all others. (b) MAE: E|y−f|\mathrm E|y-f| is finite exactly when E[y]\mathrm E[y] is; otherwise differences are finite and a median is still preferred. (c) MAPE: when E[1/y]\mathrm E[1/y] is finite, the expected MAPE is finite and minimised by a median of the actuals weighted by 1/y1/y; when E[1/y]\mathrm E[1/y] is infinite, every positive forecast has infinite expected MAPE while the forecast 0 has MAPE 100, so MAPE targets 0; when yy is 0 with positive probability, MAPE is infinite or undefined for every forecast and targets nothing.

SupportS 100C11 · formal · planned test · novelty: new, by the authors

Let the actuals be μ+ε\mu + \varepsilon, where ε\varepsilon is symmetric about 0 and its density decreases away from 0 (for example, normal). If μ>0\mu > 0, every forecast with the lowest expected sMAPE is positive; if μ<0\mu < 0, every one is negative. These forecasts stay away from 0 as μ\mu approaches 0. So as μ\mu moves from slightly below 0 to slightly above, the sMAPE target jumps from a negative forecast to a positive one.

SupportS 93C13 · descriptive · planned estimate · novelty: new, by the authors

Computed from the distributions themselves, the sMAPE target on the 34 outcome distributions of plan PL1 is the median on lognormal outcomes, lies below it when ln⁡y\ln y is skewed right and above it when skewed left (quantile levels from 0.439 to 0.556 on the unimodal positive distributions), is two forecasts, one near each mode, on equal bimodal outcomes and the larger mode's on unequal ones, lies away from zero on the side of the mean on normal outcomes straddling zero, and switches to 0 at shares of zero outcomes from 0.343 to 0.468, all below one half. On test sets the forecast sMAPE ranks first spreads widely when they are small: on lognormal outcomes with σ=1\sigma = 1, its quantile level runs from 0.16 to 0.82 (5th to 95th percentile over test sets) with 6 outcomes and from 0.47 to 0.53 with 1000.

SupportS 93C14 · descriptive · planned estimate · novelty: new, by the authors

As the spread σ\sigma of ln⁡y\ln y grows from 0.1 to 10, the expected sMAPE at the sMAPE target rises towards its bound 200 (from 7.97 to 178.2 on lognormal outcomes), the target's quantile level moves from the median to that of the mode of ln⁡y\ln y (0.372 against the mode's 0.368 at σ=10\sigma = 10 on log-Gumbel outcomes skewed right), and the ranking agreement of sMAPE, for its target against a forecast half a σ\sigma away, is lower than that of MAE: at σ=4\sigma = 4 with test sets of 12 outcomes it ranks its target first in 0.61 of the test sets, MAE in 0.82.

SupportS 99C16 · descriptive · planned estimate · novelty: new, by the authors

Computed from the distributions themselves, the msMAPE target differs little from the sMAPE target away from zero and on counts, and differs near zero: it is within 0.3% of the sMAPE target on the six distributions with median about 100, and the same as the sMAPE target on every Poisson counts distribution studied; on lognormal actuals with median 1 and σ=1\sigma = 1 it is 0.985 instead of 1, and the share of zeros at which it switches to the forecast 0 is 0.470 instead of sMAPE's 0.394, while at median 100 the two shares are 0.3937 and 0.3940.

Evidence

E1 · quant · other · cited by C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C12

Numerical checks of the formulas of the formal claims (formula checks), lab run RN8 under the formal plan that decided them; every check is within the plan's tolerances. They check the formulas on concrete numbers; the proofs are the evidence for the claims. The rows are labelled by the hypotheses of the record's first plan, whose wording kept the factor 200: H1 checks C1; H2 and H10 check C2; H3 and H9 check C3; H4 and H6 check C4; H5 checks C5; H7 checks C6; H8 checks C7; H12 checks C8; H13 checks C9; H14 checks C10; the two results E1.R1 and E1.R2 check a large-sample variance that the paper reports by simulation instead (C15).

  • R1 simulated n Var(ln sMAPE minimiser) / large-sample value, lognormal sigma=1 = 0.9944 99% confidence interval 0.9354 to 1.0598 (hypothesis: H11, test-set size: 1000, test sets: 4000)
  • R2 simulated n Var(ln sMAPE minimiser) / large-sample value, lognormal sigma=1 = 1.022 99% confidence interval 0.9396 to 1.0982 (hypothesis: H11, test-set size: 10000, test sets: 2000)
  • R3 maximum absolute difference from the ratio form = 0.0000 (hypothesis: H1)
  • R4 maximum absolute difference between the ratio and its reciprocal = 0.0000 (hypothesis: H2)
  • R5 maximum absolute difference from 200 tanh(|z|/2) = 0.0000 (hypothesis: H3)
  • R6 violations of monotonicity, concavity or the bound 200 on 0 <= |z| <= 12 = 0 (hypothesis: H3)
  • R7 violations of 0 <= 100|z| - s <= 25|z|^3/3 = 0 (hypothesis: H9)
  • R8 maximum of |s - 100|z| + 25|z|^3/3| / |z|^5 for 0.05 < |z| < 0.5 = 0.8331 (hypothesis: H9)
  • R9 violations of s(y, y+e) < s(y, y-e) = 0 (hypothesis: H10)
  • R10 maximum absolute difference between s(y, ky) and s(y, y/k) = 0.0000 (hypothesis: H10)
  • R11 maximum relative difference between the formula for S' and a numerical derivative = 0.0000 (hypothesis: H4)
  • R12 maximum distance in ln f between the minimiser and the median = 0.0000 (hypothesis: H5)
  • R13 number of local minima, lognormal sigma=1 and log-Laplace b=1 = 1 (hypothesis: H5)
  • R14 maximum balance residual at the minimiser (pull below minus pull above) = 0.0000 (hypothesis: H6)
  • R15 settings with one local minimum where the sign of D misplaces the target = 0 (hypothesis: H6)
  • R16 maximum distance between the one-step and the exact quantile level = 0.0025 (hypothesis: H6)
  • R17 2 g(m) - E[sech^2(Z/2) tanh(|Z|/2)] for the equal-mode bimodal setting (negative: median a local maximum) = -0.3475 (hypothesis: H7)
  • R18 number of local minima of the equal-mode bimodal setting = 2 (hypothesis: H7)
  • R19 difference between the two minimal expected sMAPEs, equal-mode bimodal setting = 0.0000 (hypothesis: H7)
  • R20 maximum distance between the formula and the bisected threshold = 0.0000 (hypothesis: H8)
  • R21 opposite-sign or one-zero pairs not scoring 200 = 0 (hypothesis: H8)
  • R22 relative distance between the 1/y-weighted median and the minimiser of expected MAPE = 0.0001 (hypothesis: H12)
  • R23 difference of expected squared errors minus (f1-f2)(f1+f2-2E[y]) = 0 (hypothesis: H12)
  • R24 sMAPE terms outside [0, 200] = 0 (hypothesis: H13)
  • R25 expected sMAPE at f = 1e-6 and f = 1e10 (both near 200) = 200 (hypothesis: H13)
  • R26 the proof's bound on the error times sigma^3, 600 zeta(3) max|g''|, log-Gumbel = 309.9 (hypothesis: H14)
  • R27 limit of sigma^3 times the relative efficiency for normal log-outcomes, from the proof = 7.226 (hypothesis: H14)
E1.T1 · zero threshold by setting; hypothesis: H8 (lab runs RN8) (hypothesis: H8)
Rowsettingzero threshold
r1lognormal sigma=0.30.4687
r2lognormal sigma=0.50.4469
r3lognormal sigma=10.394
r4lognormal sigma=20.3075
E1.T2 · maximum |S - (200 - 800 ln2 h)| times sigma^3, log-Gumbel right, sigma^3 times the asymptotic relative efficiency, lognormal by sigma; hypothesis: H14 (lab runs RN8) (hypothesis: H14)
Rowsigmamaximum |S - (200 - 800 ln2 h)| times sigma^3, log-Gumbel rightsigma^3 times the asymptotic relative efficiency, lognormal
r15.0264.55.613
r210.0292.36.743
r320.0300.17.099
r440.0302.17.193

E11 · quant · other · cited by C11

Numerical checks of the formulas of C11 (formula checks), lab run RN6 under the formal plan that decided C11; every check is within the plan's tolerances. They check the formulas on concrete numbers; the proof is the evidence for the claim.

E11.T1 · sign of the global minimiser of expected sMAPE, distance of the global minimiser from 0, lowest expected sMAPE term, local minima of expected sMAPE, smallest distance of any local minimiser from 0 by mean; hypothesis: H31 (lab runs RN6) (hypothesis: H31)
Rowmeansign of the global minimiser of expected sMAPEdistance of the global minimiser from 0lowest expected sMAPE termlocal minima of expected sMAPEsmallest distance of any local minimiser from 0
r1-0.5-10.98380.522720.5849
r2-0.2-10.83710.60920.679
r3-0.05-10.7730.652520.7335
r4-0.01-10.75690.66420.7491
r50.0110.75690.66420.7491
r60.0510.7730.652520.7336
r70.210.83710.60920.679
r80.510.98380.522720.5849
E11.T2 · expected sMAPE term at a forecast near 0 by mean, forecast; hypothesis: H31 (lab runs RN6) (hypothesis: H31)
Rowmeanforecastexpected sMAPE term at a forecast near 0
r1-0.5-0.0010.9948
r2-0.50.0010.9957
r3-0.2-0.0010.9946
r4-0.20.0010.995
r5-0.05-0.0010.9947
r6-0.050.0010.9948
r7-0.01-0.0010.9947
r8-0.010.0010.9948
r90.01-0.0010.9948
r100.010.0010.9947
r110.05-0.0010.9948
r120.050.0010.9947
r130.2-0.0010.995
r140.20.0010.9946
r150.5-0.0010.9957
r160.50.0010.9948

E13 · quant · other · cited by C13

Lab runs RN1 under plan PL1. Design: Selection design (D16) on 33 outcome distributions: inverse Gaussian (6), log-skewed (12) and lognormal (2), bimodal (3), straddling zero (3), zero-inflated lognormal (8). For each: the derived values (D15) of every local minimum of expected sMAPE with its quantile level and expected score; the one-step quantile level of H6 and the pull difference D at the median; the zero threshold of H8; the quantile level of the log-mode; which moments are finite (H12); the MAPE target (H12); each measure's preferred candidate (D14), ranked by finite differences where expected scores are infinite. Candidates: quantiles at levels 0.01 to 0.99, the mean (when finite), 0, the log-mode, the MAPE minimiser (when it exists and is not 0) and every local minimiser of expected sMAPE, merged when within 1e-4 (relative). Test sets of n = 6, 8, 12, 13, 14, 18, 24, 36, 48, 100, 250, 500, 1000 and 10000 outcomes, 1000 test sets per size, seed 3 with the setting's index and n. Measures RMSE, MAE, MAPE, sMAPE on every setting. python3 claimset-code/smape-target/targets.py --seed 3 --reps 1000 (numpy 2.5.3, matplotlib 3.11.2). Analysis: Per distribution: the derived values listed in the design. Per distribution, measure and n, over test sets, of the candidate ranked first: its quantile level's mean with a 99% confidence interval and 5th, 25th, 50th, 75th and 95th percentiles; the share choosing each candidate; the share choosing the preferred candidate with its 99% Wilson interval; the number of test sets in which every candidate scores infinite. Figures: one per distribution, a panel per measure, violins of the chosen quantile levels over n with boxplots and the mean, lines at every measure's preferred candidate and at further local minima of expected sMAPE, and a note in the panel of a measure that targets nothing. These are compared with what H4 to H8, H12 and H13 predict; no test is made and no threshold is set.

inverse Gaussian shape=1000: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is sprea…
A9 inverse Gaussian shape=1000: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=1000.png).
inverse Gaussian shape=200: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread…
A10 inverse Gaussian shape=200: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=200.png).
inverse Gaussian shape=100: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread…
A11 inverse Gaussian shape=100: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=100.png).
inverse Gaussian shape=50: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread…
A12 inverse Gaussian shape=50: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=50.png).
inverse Gaussian shape=20: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread…
A13 inverse Gaussian shape=20: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=20.png).
inverse Gaussian shape=10: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread…
A14 inverse Gaussian shape=10: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=10.png).
log-skewed left sd=0.5 skew=-2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is sp…
A15 log-skewed left sd=0.5 skew=-2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=0.5_skew=-2.png).
log-skewed left sd=0.5 skew=-1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is sp…
A16 log-skewed left sd=0.5 skew=-1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=0.5_skew=-1.png).
log-skewed left sd=0.5 skew=-0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is…
A17 log-skewed left sd=0.5 skew=-0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=0.5_skew=-0.5.png).
log-skewed right sd=0.5 skew=+2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is s…
A18 log-skewed right sd=0.5 skew=+2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=0.5_skew=_2.png).
log-skewed right sd=0.5 skew=+1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is s…
A19 log-skewed right sd=0.5 skew=+1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=0.5_skew=_1.png).
log-skewed right sd=0.5 skew=+0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is…
A20 log-skewed right sd=0.5 skew=+0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=0.5_skew=_0.5.png).
lognormal sigma=0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1…
A21 lognormal sigma=0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=0.5.png).
log-skewed left sd=1 skew=-2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spre…
A22 log-skewed left sd=1 skew=-2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=1_skew=-2.png).
log-skewed left sd=1 skew=-1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spre…
A23 log-skewed left sd=1 skew=-1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=1_skew=-1.png).
log-skewed left sd=1 skew=-0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is sp…
A24 log-skewed left sd=1 skew=-0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=1_skew=-0.5.png).
log-skewed right sd=1 skew=+2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spr…
A25 log-skewed right sd=1 skew=+2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=1_skew=_2.png).
log-skewed right sd=1 skew=+1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spr…
A26 log-skewed right sd=1 skew=+1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=1_skew=_1.png).
log-skewed right sd=1 skew=+0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is s…
A27 log-skewed right sd=1 skew=+0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=1_skew=_0.5.png).
lognormal sigma=1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 100…
A28 lognormal sigma=1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=1.png).
bimodal equal modes: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1…
A29 bimodal equal modes: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/bimodal_equal_modes.png).
bimodal left mode higher: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread o…
A30 bimodal left mode higher: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/bimodal_left_mode_higher.png).
bimodal right mode higher: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread…
A31 bimodal right mode higher: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/bimodal_right_mode_higher.png).
normal straddling zero mean=0: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spr…
A32 normal straddling zero mean=0: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/normal_straddling_zero_mean=0.png).
normal straddling zero mean=0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is s…
A33 normal straddling zero mean=0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/normal_straddling_zero_mean=0.5.png).
normal straddling zero mean=1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spr…
A34 normal straddling zero mean=1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/normal_straddling_zero_mean=1.png).
lognormal sigma=0.5 with zeros p0=0.25: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks fir…
A35 lognormal sigma=0.5 with zeros p0=0.25: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=0.5_with_zeros_p0=0.25.png).
lognormal sigma=0.5 with zeros p0=0.42: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks fir…
A36 lognormal sigma=0.5 with zeros p0=0.42: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=0.5_with_zeros_p0=0.42.png).
lognormal sigma=0.5 with zeros p0=0.47: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks fir…
A37 lognormal sigma=0.5 with zeros p0=0.47: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=0.5_with_zeros_p0=0.47.png).
lognormal sigma=0.5 with zeros p0=0.6: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks firs…
A38 lognormal sigma=0.5 with zeros p0=0.6: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=0.5_with_zeros_p0=0.6.png).
lognormal sigma=1 with zeros p0=0.25: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first…
A39 lognormal sigma=1 with zeros p0=0.25: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=1_with_zeros_p0=0.25.png).
lognormal sigma=1 with zeros p0=0.37: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first…
A40 lognormal sigma=1 with zeros p0=0.37: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=1_with_zeros_p0=0.37.png).
lognormal sigma=1 with zeros p0=0.42: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first…
A41 lognormal sigma=1 with zeros p0=0.42: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=1_with_zeros_p0=0.42.png).
lognormal sigma=1 with zeros p0=0.6: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first…
A42 lognormal sigma=1 with zeros p0=0.6: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=1_with_zeros_p0=0.6.png).
E13.T1 · The sMAPE target and related values derived from each distribution: the lowest local minimum of expected sMAPE (rank 1), its quantile level and expected sMAPE, its one-step approximation, the pull from above minus the pull from below at the median, the share of zeros at which the target switches to 0, the quantile level of the log-mode, and the MAPE target
Rowsettingderived local minimum of expected sMAPE (rank: 1)quantile level of a derived local minimum of expected sMAPE (rank: 1)expected sMAPE at a derived local minimum (rank: 1)one-step quantile level of the sMAPE targetpull from above minus pull from below at the medianzero threshold of sMAPEquantile level of the log-modemedian of the outcome distributionmean of the outcome distribution
r1inverse Gaussian shape=100095.270.499924.360.4999-0.00010.46760.49895.27100
r2inverse Gaussian shape=20080.140.497848.680.4978-0.00190.43070.482280.43100
r3inverse Gaussian shape=10066.530.492961.970.4929-0.00520.40830.459767.58100
r4inverse Gaussian shape=5048.860.48275.050.482-0.01140.38450.423351.42100
r5inverse Gaussian shape=2025.90.457888.960.457-0.02240.3570.372630.11100
r6inverse Gaussian shape=1014.020.438795.750.4362-0.030.34260.347117.85100
r7log-skewed left sd=0.5 skew=-21190.521132.310.52110.01970.45611116.6109.9
r8log-skewed left sd=0.5 skew=-1110.20.51336.740.5130.01170.44940.6472108.5111.3
r9log-skewed left sd=0.5 skew=-0.5105.10.506837.960.50680.00610.44760.5681104.2112.2
r10log-skewed right sd=0.5 skew=+284.020.478932.310.4789-0.01970.4561085.78121.3
r11log-skewed right sd=0.5 skew=+190.720.48736.740.487-0.01170.44940.352892.13116.3
r12log-skewed right sd=0.5 skew=+0.595.130.493237.960.4932-0.00610.44760.431995.93114.6
r13lognormal sigma=0.51000.538.380.50.00000.44690.5100113.3
r14log-skewed left sd=1 skew=-2151.20.556257.040.55720.04560.41681135.9135.9
r15log-skewed left sd=1 skew=-1128.30.536266.420.53650.02610.40040.6472117.8146
r16log-skewed left sd=1 skew=-0.5113.90.519169.050.51920.01330.39570.5681108.7153.7
r17log-skewed right sd=1 skew=+266.140.443857.040.4428-0.04560.4168073.58
r18log-skewed right sd=1 skew=+177.960.463866.420.4635-0.02610.40040.352884.88216.5
r19log-skewed right sd=1 skew=+0.587.790.480969.050.4808-0.01330.39570.431992.03182.7
r20lognormal sigma=11000.569.940.50.00000.3940.5100164.9
r21bimodal equal modes43.490.374283.840.50.00000.36740.25100159.2
r22bimodal left mode higher39.840.406365.210.4042-0.08390.40260.32544.22122.8
r23bimodal right mode higher2510.593765.210.59580.08390.40260.675226.1195.6
r24normal straddling zero mean=0-0.7530.2257133.40.00000
r25normal straddling zero mean=0.50.98380.6858104.50.50.5
r26normal straddling zero mean=11.2870.612878.5111
r27lognormal sigma=0.5 with zeros p0=0.251000.62578.780.50.00000.44690.62580.6284.99
r28lognormal sigma=0.5 with zeros p0=0.421000.71106.30.50.00000.44690.7157.9965.72
r29lognormal sigma=0.5 with zeros p0=0.4700.471060.50.00000.44690.73545.360.06
r30lognormal sigma=0.5 with zeros p0=0.600.6800.50.00000.44690.8045.33
r31lognormal sigma=1 with zeros p0=0.251000.625102.50.50.00000.3940.62565123.7
r32lognormal sigma=1 with zeros p0=0.371000.685118.10.50.00000.3940.68544.08103.9
r33lognormal sigma=1 with zeros p0=0.4200.421160.50.00000.3940.7133.6395.63
r34lognormal sigma=1 with zeros p0=0.600.6800.50.00000.3940.8065.95
E13.T2 · The second local minimum of expected sMAPE, where there is one (rank 2)
Rowsettingderived local minimum of expected sMAPE (rank: 2)quantile level of a derived local minimum of expected sMAPE (rank: 2)expected sMAPE at a derived local minimum (rank: 2)
r1bimodal equal modes229.90.625883.84
r2normal straddling zero mean=00.7530.7743133.4
r3normal straddling zero mean=0.5-0.58490.139159.9
r4normal straddling zero mean=1-0.46490.0715179.8
r5lognormal sigma=0.5 with zeros p0=0.2500.25150
r6lognormal sigma=0.5 with zeros p0=0.4200.42116
r7lognormal sigma=0.5 with zeros p0=0.471000.735114.3
r8lognormal sigma=0.5 with zeros p0=0.61000.8135.4
r9lognormal sigma=1 with zeros p0=0.2500.25150
r10lognormal sigma=1 with zeros p0=0.3700.37126
r11lognormal sigma=1 with zeros p0=0.421000.71124.6
r12lognormal sigma=1 with zeros p0=0.61000.8148
E13.T3 · Each measure's preferred candidate (the candidate forecast with the lowest expected score): its quantile level, and the number of candidates whose expected score is infinite
Rowsettingquantile level of the preferred candidate (measure: RMSE)quantile level of the preferred candidate (measure: MAE)quantile level of the preferred candidate (measure: MAPE)quantile level of the preferred candidate (measure: sMAPE)candidates whose expected score is infinite (measure: RMSE)candidates whose expected score is infinite (measure: MAPE)
r1inverse Gaussian shape=10000.56160.49990.37860.499900
r2inverse Gaussian shape=2000.62770.50.26590.497800
r3inverse Gaussian shape=1000.66810.50.21580.492900
r4inverse Gaussian shape=500.71380.50.17730.48200
r5inverse Gaussian shape=200.77680.50.14740.457800
r6inverse Gaussian shape=100.82190.50.1360.438700
r7log-skewed left sd=0.5 skew=-20.44440.50.250.521100
r8log-skewed left sd=0.5 skew=-10.52150.50.27990.51300
r9log-skewed left sd=0.5 skew=-0.50.56030.50.29410.506800
r10log-skewed right sd=0.5 skew=+20.750.50.370.47891020
r11log-skewed right sd=0.5 skew=+10.67480.50.33880.48700
r12log-skewed right sd=0.5 skew=+0.50.63680.50.32340.493200
r13lognormal sigma=0.50.59870.50.30850.500
r14log-skewed left sd=1 skew=-20.50.500.55620101
r15log-skewed left sd=1 skew=-10.59280.50.06550.536200
r16log-skewed left sd=1 skew=-0.50.64130.50.11550.519100
r17log-skewed right sd=1 skew=+20.50.29290.44381030
r18log-skewed right sd=1 skew=+10.80340.50.23140.46381040
r19log-skewed right sd=1 skew=+0.50.74460.50.19680.480900
r20lognormal sigma=10.69150.50.15870.500
r21bimodal equal modes0.50810.50.23410.374200
r22bimodal left mode higher0.65030.50.28410.406300
r23bimodal right mode higher0.41110.50.18480.593700
r24normal straddling zero mean=00.50.50.50.77430100
r25normal straddling zero mean=0.50.50.50.30850.68580101
r26normal straddling zero mean=10.50.50.15870.61280101
r27lognormal sigma=0.5 with zeros p0=0.250.52930.50.625077
r28lognormal sigma=0.5 with zeros p0=0.420.53640.50.71059
r29lognormal sigma=0.5 with zeros p0=0.470.55160.50.47055
r30lognormal sigma=0.5 with zeros p0=0.60.62270.60.6041
r31lognormal sigma=1 with zeros p0=0.250.68810.50.625077
r32lognormal sigma=1 with zeros p0=0.370.69450.50.685065
r33lognormal sigma=1 with zeros p0=0.420.69970.50.42059
r34lognormal sigma=1 with zeros p0=0.60.73540.60.6041
E13.T4 · The candidates a measure ranks first over test sets, for four of the distributions at test-set sizes 6, 100 and 1000: the mean and the 5th, 50th and 95th percentiles of their quantile levels, the share of test sets choosing the preferred candidate, and (MAPE) the test sets in which every candidate's score is infinite; every distribution and size is in the run's CSV and its figure
Rowsettingmeasuretest-set sizemean quantile level of the chosen candidatespercentile of the chosen candidates' quantile levels (percentile: 5)percentile of the chosen candidates' quantile levels (percentile: 50)percentile of the chosen candidates' quantile levels (percentile: 95)share of test sets choosing the preferred candidatetest sets in which every candidate's score is infinite
r1lognormal sigma=1RMSE60.62870.34950.640.880.012
r2lognormal sigma=1RMSE1000.68680.610.680.770.041
r3lognormal sigma=1RMSE10000.69110.670.69150.71050.14
r4lognormal sigma=1MAE60.49870.22640.5050.75660.0222
r5lognormal sigma=1MAE1000.49870.420.50.580.0733
r6lognormal sigma=1MAE10000.50020.470.50.530.256
r7lognormal sigma=1MAPE60.2370.01950.210.58050.006
r8lognormal sigma=1MAPE1000.16140.090.15870.240.041
r9lognormal sigma=1MAPE10000.15940.130.160.190.123
r10lognormal sigma=1sMAPE60.49880.160.510.820.007
r11lognormal sigma=1sMAPE1000.49910.40.50.60.063
r12lognormal sigma=1sMAPE10000.50030.470.50.530.201
r13inverse Gaussian shape=10RMSE60.70760.40.730.940.014
r14inverse Gaussian shape=10RMSE1000.81290.740.820.870.053
r15inverse Gaussian shape=10RMSE10000.82030.80.82190.840.175
r16inverse Gaussian shape=10MAE60.490.23550.480.76770.0192
r17inverse Gaussian shape=10MAE1000.49790.4150.50.580.09
r18inverse Gaussian shape=10MAE10000.50.470.50.520.25
r19inverse Gaussian shape=10MAPE60.19910.010.170.490.008
r20inverse Gaussian shape=10MAPE1000.13970.080.1360.20.049
r21inverse Gaussian shape=10MAPE10000.13690.120.1360.160.157
r22inverse Gaussian shape=10sMAPE60.45760.13980.43870.83050.008
r23inverse Gaussian shape=10sMAPE1000.43920.320.43870.560.025
r24inverse Gaussian shape=10sMAPE10000.4390.40.43870.480.084
r25bimodal equal modesRMSE60.53550.490.50810.690.223
r26bimodal equal modesRMSE1000.51210.50810.50810.530.53
r27bimodal equal modesRMSE10000.50880.50810.50810.510.65
r28bimodal equal modesMAE60.49630.2260.49670.760.017
r29bimodal equal modesMAE1000.50120.420.5060.58520.0431
r30bimodal equal modesMAE10000.49940.470.50.530.06
r31bimodal equal modesMAPE60.2550.040.250.470.011
r32bimodal equal modesMAPE1000.23420.180.23410.290.056
r33bimodal equal modesMAPE10000.23440.210.23410.250.164
r34bimodal equal modessMAPE60.49520.140.470.840.011
r35bimodal equal modessMAPE1000.50650.310.5850.70.024
r36bimodal equal modessMAPE10000.49980.350.60.650.077
r37lognormal sigma=1 with zeros p0=0.42RMSE60.65450.440.650.890.013
r38lognormal sigma=1 with zeros p0=0.42RMSE1000.69420.620.690.770.052
r39lognormal sigma=1 with zeros p0=0.42RMSE10000.6990.680.69970.720.124
r40lognormal sigma=1 with zeros p0=0.42MAE60.53750.420.510.75020.0227
r41lognormal sigma=1 with zeros p0=0.42MAE1000.49880.420.4950.5850.0754
r42lognormal sigma=1 with zeros p0=0.42MAE10000.49950.470.50.530.248
r43lognormal sigma=1 with zeros p0=0.42MAPE60.55240.430.560.688963
r44lognormal sigma=1 with zeros p0=0.42MAPE1001000
r45lognormal sigma=1 with zeros p0=0.42MAPE10001000
r46lognormal sigma=1 with zeros p0=0.42sMAPE60.55330.420.420.890.539
r47lognormal sigma=1 with zeros p0=0.42sMAPE1000.50860.420.420.750.695
r48lognormal sigma=1 with zeros p0=0.42sMAPE10000.43460.420.420.4330.95
E13.T5 · The MAPE target derived from each distribution (a median of the actuals weighted by 1/y), where it is finite
Rowsettingderived MAPE target
r1inverse Gaussian shape=100086.51
r2inverse Gaussian shape=20053.48
r3inverse Gaussian shape=10034.69
r4inverse Gaussian shape=5019.7
r5inverse Gaussian shape=208.334
r6inverse Gaussian shape=104.21
r7log-skewed left sd=0.5 skew=-282.43
r8log-skewed left sd=0.5 skew=-179.93
r9log-skewed left sd=0.5 skew=-0.578.8
r10log-skewed right sd=0.5 skew=+276.42
r11log-skewed right sd=0.5 skew=+176.67
r12log-skewed right sd=0.5 skew=+0.577.17
r13lognormal sigma=0.577.88
r14log-skewed left sd=1 skew=-20
r15log-skewed left sd=1 skew=-118.79
r16log-skewed left sd=1 skew=-0.529.44
r17log-skewed right sd=1 skew=+252.02
r18log-skewed right sd=1 skew=+146.02
r19log-skewed right sd=1 skew=+0.542.04
r20lognormal sigma=136.79
r21bimodal equal modes36.06
r22bimodal left mode higher35.36
r23bimodal right mode higher37.44
r24normal straddling zero mean=00
r25normal straddling zero mean=0.50
r26normal straddling zero mean=10
E13.T6 · The value of each measure's preferred candidate, for the four distributions of T4
Rowsettingmeasurevalue of the preferred candidate
r1lognormal sigma=1RMSE164.9
r2lognormal sigma=1MAE100
r3lognormal sigma=1MAPE36.79
r4lognormal sigma=1sMAPE100
r5inverse Gaussian shape=10RMSE100
r6inverse Gaussian shape=10MAE17.85
r7inverse Gaussian shape=10MAPE4.21
r8inverse Gaussian shape=10sMAPE14.02
r9bimodal equal modesRMSE159.2
r10bimodal equal modesMAE100
r11bimodal equal modesMAPE36.06
r12bimodal equal modessMAPE43.49
r13lognormal sigma=1 with zeros p0=0.42RMSE95.63
r14lognormal sigma=1 with zeros p0=0.42MAE33.63
r15lognormal sigma=1 with zeros p0=0.42sMAPE0

E14 · quant · other · cited by C14

Lab runs RN2 under plan PL4. Design: Flatness design (D18) on 30 distributions (D27): lognormal, log-Gumbel right and log-Gumbel left, with sigma 0.1, 0.25, 0.5, 1, 1.5, 2, 3, 4, 6, 10. Derived (D15) from each: the quantile levels of the sMAPE target and of the log-mode; expected sMAPE at the target and the approximation of H14 there; the number of local minima; for each measure (RMSE, MAE, MAPE, sMAPE) and forecasts e^(k sigma) times its own target (k = -1, -0.5, +0.5, +1: half and one standard deviation of ln y away), the excess expected score over the target's, relative to the target's, where both are finite, and sMAPE's absolute excess; expected sMAPE at quantile levels 0.01 to 0.99. On test sets (10000 per case, seed 5 with the setting's index): the ranking agreement (D20) of each measure between its target and those four forecasts, on the lognormal family (on which every measure has a target), at n = 12, 48, 250 and 1000; the quantile level of the sMAPE minimiser of a test set at n = 48 and 1000 on every distribution. python3 claimset-code/smape-target/flatness.py --seed 5 --reps 10000 (numpy 2.5.3, matplotlib 3.11.2). Analysis: The derived values listed in the design, with 'none' and its reason where a measure targets nothing or its expected score is infinite; the ranking agreement with its 99% Wilson interval; the mean quantile level of the test sets' sMAPE minimisers with a 99% confidence interval and its 5th, 25th, 50th, 75th and 95th percentiles. Figures: the target's quantile level over sigma beside the log-mode's and the median; expected sMAPE over the quantile level of the forecast, a line per sigma; the relative excess over sigma by measure; ranking agreement over sigma by measure and n; the spread of the test sets' sMAPE minimisers over sigma. Compared with H14 (i) and (ii), and with the other measures; no test is made and no threshold is set.

The quantile level of the sMAPE target against the spread sigma of ln y, for lognormal and log-Gumbel outcomes, beside the level of the log-mode: the target mo…
A44 The quantile level of the sMAPE target against the spread sigma of ln y, for lognormal and log-Gumbel outcomes, beside the level of the log-mode: the target moves from the median to the mode of ln y as the spread grows. Drawn by lab run RN2 from its own results (lab/runs/RN2/figures/flatness_target.png).
Expected sMAPE against the quantile level of the forecast, one curve per spread sigma, for each family: the curve flattens towards 200 as sigma grows.
A45 Expected sMAPE against the quantile level of the forecast, one curve per spread sigma, for each family: the curve flattens towards 200 as sigma grows. Drawn by lab run RN2 from its own results (lab/runs/RN2/figures/flatness_expected_sMAPE.png).
Lognormal outcomes: how much worse a forecast half a sigma or one sigma away from each measure's target scores, as a share of the score at the target, against…
A46 Lognormal outcomes: how much worse a forecast half a sigma or one sigma away from each measure's target scores, as a share of the score at the target, against sigma. Drawn by lab run RN2 from its own results (lab/runs/RN2/figures/flatness_sensitivity.png).
Ranking agreement: the share of test sets that prefer each measure's target to a forecast away from it, against sigma, for several test-set sizes.
A47 Ranking agreement: the share of test sets that prefer each measure's target to a forecast away from it, against sigma, for several test-set sizes. Drawn by lab run RN2 from its own results (lab/runs/RN2/figures/flatness_agreement.png).
Where the sMAPE minimiser of a test set falls, and how widely it spreads: its quantile level over test sets against sigma, for each family.
A48 Where the sMAPE minimiser of a test set falls, and how widely it spreads: its quantile level over test sets against sigma, for each family. Drawn by lab run RN2 from its own results (lab/runs/RN2/figures/flatness_minimiser.png).
E14.T1 · The sMAPE target as the spread of ln y grows: its quantile level beside the log-mode's, the expected sMAPE there and its large-spread approximation
Rowsettingquantile level of the sMAPE targetquantile level of the log-modeexpected sMAPE at the targetlarge-spread approximation of expected sMAPE at the targetlocal minima of expected sMAPE
r1lognormal sigma=0.10.50.57.966-20121
r2log-Gumbel right sigma=0.10.49890.36799.643-17241
r3log-Gumbel left sigma=0.10.50110.63219.643-17241
r4lognormal sigma=0.250.50.519.74-684.91
r5log-Gumbel right sigma=0.250.4940.367923.66-572.71
r6log-Gumbel left sigma=0.250.5060.632123.66-572.71
r7lognormal sigma=0.50.50.538.38-242.41
r8log-Gumbel right sigma=0.50.48110.367944.85-190.41
r9log-Gumbel left sigma=0.50.51890.632144.85-190.41
r10lognormal sigma=10.50.569.94-21.221
r11log-Gumbel right sigma=10.45440.367977.991.2461
r12log-Gumbel left sigma=10.54560.632177.991.2461
r13lognormal sigma=1.50.50.593.6652.521
r14log-Gumbel right sigma=1.50.43370.3679101.266.061
r15log-Gumbel left sigma=1.50.56630.6321101.266.061
r16lognormal sigma=20.50.5111.289.391
r17log-Gumbel right sigma=20.41870.3679117.998.931
r18log-Gumbel left sigma=20.58130.6321117.998.931
r19lognormal sigma=30.50.5134.4126.31
r20log-Gumbel right sigma=30.39990.3679139.5132.31
r21log-Gumbel left sigma=30.60010.6321139.5132.31
r22lognormal sigma=40.50.5148.5144.71
r23log-Gumbel right sigma=40.38940.3679152.5149.11
r24log-Gumbel left sigma=40.61060.6321152.5149.11
r25lognormal sigma=60.50.5164.4163.11
r26log-Gumbel right sigma=60.37920.3679167.11661
r27log-Gumbel left sigma=60.62080.6321167.11661
r28lognormal sigma=100.50.5178.2177.91
r29log-Gumbel right sigma=100.37240.3679179.9179.61
r30log-Gumbel left sigma=100.62760.6321179.9179.61
E14.T2 · Ranking agreement: the share of test sets that give a measure's target a lower score than a forecast half a sigma above it, for three of the settings; every setting, offset and size is in the run's CSV and its figure
Rowsettingtest-set sizeranking agreement (measure: RMSE, offset in sigma: 0.5)ranking agreement (measure: MAE, offset in sigma: 0.5)ranking agreement (measure: MAPE, offset in sigma: 0.5)ranking agreement (measure: sMAPE, offset in sigma: 0.5)
r1lognormal sigma=0.5120.82820.77720.73960.7557
r2lognormal sigma=0.5480.96590.94210.91180.9233
r3lognormal sigma=0.52500.99980.99990.99940.9994
r4lognormal sigma=0.510001111
r5lognormal sigma=4120.98260.82260.00120.605
r6lognormal sigma=4480.97920.97050.00590.6989
r7lognormal sigma=42500.975910.03370.8823
r8lognormal sigma=410000.977110.09280.9909
E14.T3 · How much worse a forecast half a sigma or one sigma below or above the target scores, as a share of the score at the target, for lognormal outcomes
Rowsettingoffset in sigmaexcess expected score over the target, relative (measure: RMSE)excess expected score over the target, relative (measure: MAE)excess expected score over the target, relative (measure: MAPE)excess expected score over the target, relative (measure: sMAPE)excess expected sMAPE over the target (measure: sMAPE)
r1lognormal sigma=0.5-1.00.2430.30830.30830.421216.17
r2lognormal sigma=0.5-0.50.08270.09550.09550.11314.34
r3lognormal sigma=0.50.50.13310.13310.13310.11314.34
r4lognormal sigma=0.51.00.57530.5940.5940.421216.17
r5lognormal sigma=2-1.00.00690.03170.03170.227925.34
r6lognormal sigma=2-0.50.00370.01470.01470.06487.207
r7lognormal sigma=20.50.02720.05530.05530.06487.207
r8lognormal sigma=21.00.32730.43050.43040.227925.34
r9lognormal sigma=6-1.000.00000.00000.08113.32
r10lognormal sigma=6-0.500.00000.00000.02393.937
r11lognormal sigma=60.50.00000.00000.00000.02393.937
r12lognormal sigma=61.00.00000.00000.00000.08113.32
E14.T4 · Where the sMAPE minimiser of a test set falls: the mean and the 5th, 50th and 95th percentiles of its quantile level over test sets
Rowsettingtest-set sizemean quantile level of the sMAPE minimiser of a test setpercentile of the quantile level of the sMAPE minimiser of a test set (percentile: 5)percentile of the quantile level of the sMAPE minimiser of a test set (percentile: 50)percentile of the quantile level of the sMAPE minimiser of a test set (percentile: 95)
r1lognormal sigma=0.1480.50040.37640.49980.6251
r2lognormal sigma=0.110000.50.47420.49970.5263
r3log-Gumbel right sigma=0.1480.48960.36970.48930.6081
r4log-Gumbel right sigma=0.110000.49860.47240.49870.5245
r5log-Gumbel left sigma=0.1480.50910.38970.50930.6263
r6log-Gumbel left sigma=0.110000.50120.47520.50130.5275
r7lognormal sigma=0.25480.50.37620.4990.6242
r8lognormal sigma=0.2510000.49970.47290.49990.526
r9log-Gumbel right sigma=0.25480.48970.37410.48920.6057
r10log-Gumbel right sigma=0.2510000.49420.46820.49410.5206
r11log-Gumbel left sigma=0.25480.51160.39290.51160.6293
r12log-Gumbel left sigma=0.2510000.50620.48010.50640.5321
r13lognormal sigma=0.5480.50030.37760.50020.624
r14lognormal sigma=0.510000.49990.47280.49990.5274
r15log-Gumbel right sigma=0.5480.4830.36030.4820.6085
r16log-Gumbel right sigma=0.510000.48090.45290.4810.5087
r17log-Gumbel left sigma=0.5480.51750.39350.5180.6407
r18log-Gumbel left sigma=0.510000.51870.49080.51890.5467
r19lognormal sigma=1480.49960.3590.49950.6414
r20lognormal sigma=110000.50010.46810.50020.5316
r21log-Gumbel right sigma=1480.45580.31640.45450.6006
r22log-Gumbel right sigma=110000.45460.42290.45440.4874
r23log-Gumbel left sigma=1480.54380.39570.54610.684
r24log-Gumbel left sigma=110000.54580.51310.54580.5776
r25lognormal sigma=1.5480.50020.33640.49960.6636
r26lognormal sigma=1.510000.49970.46130.49950.5376
r27log-Gumbel right sigma=1.5480.43670.27550.43250.6074
r28log-Gumbel right sigma=1.510000.43410.3950.4340.4726
r29log-Gumbel left sigma=1.5480.56170.38740.56370.7247
r30log-Gumbel left sigma=1.510000.5660.52680.56660.604
r31lognormal sigma=2480.50240.31060.50190.6923
r32lognormal sigma=210000.50010.45360.50010.5468
r33log-Gumbel right sigma=2480.42540.250.41940.6192
r34log-Gumbel right sigma=210000.41910.37320.41890.4667
r35log-Gumbel left sigma=2480.57190.37630.57720.751
r36log-Gumbel left sigma=210000.58080.53330.58150.6256
r37lognormal sigma=3480.49770.27560.49920.7205
r38lognormal sigma=310000.49940.43460.49910.5664
r39log-Gumbel right sigma=3480.41540.21210.40740.6461
r40log-Gumbel right sigma=310000.40130.33850.40050.4676
r41log-Gumbel left sigma=3480.58450.35260.59170.7901
r42log-Gumbel left sigma=310000.59860.53070.59980.6617
r43lognormal sigma=4480.49850.25810.49840.7413
r44lognormal sigma=410000.49980.41340.49990.5844
r45log-Gumbel right sigma=4480.40880.18740.40030.6567
r46log-Gumbel right sigma=410000.39170.31320.38950.4769
r47log-Gumbel left sigma=4480.58870.33870.59730.8117
r48log-Gumbel left sigma=410000.60820.52070.61060.6879
r49lognormal sigma=6480.49870.23820.49760.7646
r50lognormal sigma=610000.50040.38660.50130.613
r51log-Gumbel right sigma=6480.40830.16840.39920.6722
r52log-Gumbel right sigma=610000.38390.27970.38110.4981
r53log-Gumbel left sigma=6480.58990.31670.59860.8314
r54log-Gumbel left sigma=610000.61570.50360.61870.7188
r55lognormal sigma=10480.49990.22160.50040.7798
r56lognormal sigma=1010000.50010.35940.49970.6419
r57log-Gumbel right sigma=10480.40710.1530.39560.698
r58log-Gumbel right sigma=1010000.38050.25290.37670.5195
r59log-Gumbel left sigma=10480.59150.3040.59860.8482
r60log-Gumbel left sigma=1010000.61880.48130.62270.7469

E15 · quant · other · cited by C15

Lab runs RN7 under plan PL2. Design: Efficiency design (D17) on 16 log-symmetric distributions (D26): lognormal sigma and log-Laplace b, each 0.25, 0.5, 1, 1.5, 2, 3, 4, 6. Test sets of n = 6, 8, 12, 13, 14, 18, 24, 36, 48, 100, 250, 500, 1000 and 10000 outcomes, 10000 test sets per size, seed 4 with the setting's index and n. In each test set: the sample median and the exact sMAPE minimiser (D8). From each distribution, derived (D15): the large-sample n Var of ln of each, from H11, and their ratio. python3 claimset-code/smape-target/efficiency.py --seed 4 --reps 10000 (numpy 2.5.3, matplotlib 3.11.2). Analysis: Per distribution and n, over test sets: the variances of ln(sample median) and ln(sMAPE minimiser); the relative efficiency (D9) with a 99% bootstrap percentile interval (2000 resamples of test sets); the ratio of their mean squared errors about the true median; the mean of ln(sMAPE minimiser) minus ln(median). Per distribution: the asymptotic relative efficiency. Figure: relative efficiency over n, a panel per distribution, with its band, the asymptotic value dashed and 1 dotted. Compared with H11 and H14 (iii); no test is made and no threshold is set.

lognormal outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its 9…
A50 lognormal outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its 99% band and the large-sample value as a dashed line; below 1 the sMAPE minimiser varies more than the sample median. Drawn by lab run RN7 from its own results (lab/runs/RN7/figures/efficiency_lognormal.png).
log-Laplace outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its…
A51 log-Laplace outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its 99% band and the large-sample value as a dashed line; below 1 the sMAPE minimiser varies more than the sample median. Drawn by lab run RN7 from its own results (lab/runs/RN7/figures/efficiency_log-Laplace.png).
E15.T1 · Relative efficiency of the sMAPE minimiser of a test set against the sample median, at test-set sizes 6, 12, 100, 1000 and 10000, beside its large-sample value
Rowsettingrelative efficiency (test-set size: 6)relative efficiency (test-set size: 12)relative efficiency (test-set size: 100)relative efficiency (test-set size: 1000)relative efficiency (test-set size: 10000)asymptotic relative efficiency
r1lognormal sigma=0.250.60590.70260.89960.96620.97010.9696
r2lognormal sigma=0.50.6110.70830.88020.8850.89060.8881
r3lognormal sigma=10.60670.65550.65240.66590.65650.6601
r4lognormal sigma=1.50.54840.51230.45950.46330.450.454
r5lognormal sigma=20.5010.43920.3230.30770.30430.3073
r6lognormal sigma=30.43810.36460.20580.15570.15130.1474
r7lognormal sigma=40.41480.32720.15610.0890.07890.0778
r8lognormal sigma=60.39330.30270.12290.05080.03170.0279
r9log-Laplace b=0.250.70490.82680.97690.99180.99690.9969
r10log-Laplace b=0.50.69890.8190.94480.97790.97530.9765
r11log-Laplace b=10.6670.68870.84790.87630.88530.8953
r12log-Laplace b=1.50.57660.59470.69890.77890.78040.8035
r13log-Laplace b=20.51790.50360.6120.6830.70790.7206
r14log-Laplace b=30.46720.43650.45860.52570.56280.5902
r15log-Laplace b=40.43530.37640.34960.42670.48030.4968
r16log-Laplace b=60.41670.34920.26670.28960.34410.3753
E15.T2 · Bias and variance at test-set sizes 12 and 1000, for spreads 0.25, 1 and 6: the mean of ln(sMAPE minimiser) minus ln(median), and the variances of ln(sample median) and ln(sMAPE minimiser); every setting and size is in the run's CSV
Rowsettingtest-set sizemean of ln(sMAPE minimiser) minus ln(median)variance of ln(sample median)variance of ln(sMAPE minimiser)ratio of mean squared errors about the true median
r1lognormal sigma=0.25120.00080.00720.01030.7026
r2lognormal sigma=0.2510000.00000.00010.00010.9662
r3lognormal sigma=1120.00080.11590.17680.6555
r4lognormal sigma=110000.00010.00160.00240.6659
r5lognormal sigma=6120.00214.28714.170.3027
r6lognormal sigma=61000-0.02090.05711.1260.0507
r7log-Laplace b=0.2512-0.00040.00730.00880.8268
r8log-Laplace b=0.2510000.00020.00010.00010.9918
r9log-Laplace b=1120.00140.11910.1730.6887
r10log-Laplace b=110000.00030.00110.00120.8764
r11log-Laplace b=6120.02124.26712.220.3492
r12log-Laplace b=610000.0030.03770.13030.2896

E16 · quant · other · cited by C16

Lab runs RN10 under plan PL11. Design: Exact computation, with no sampling, of the forecasts that minimise the expected msMAPE term and the expected sMAPE term (claimset-code/smape-target/msmape_targets.py, run once): by Gauss-Legendre quadrature of the density of ln y for continuous distributions (with zeros added as a point mass) and by sums for counts, minimised over forecasts f >= 0 on a grid of 0 and 4000 log-spaced points refined by golden section. Settings: away from zero, lognormal sigma 0.5 and 1, inverse Gaussian shape 10 and 100 and log-skewed sd 1 with skewness -1 and +1, all at median about 100; near zero, lognormal sigma 1 at median 1 and at median 100 with shares of zeros 0, 0.2, 0.4, 0.6 and 0.8; counts, Poisson with means 0.3, 0.5, 1, 1.5, 3 and 10. For the lognormal sigma 1 at medians 1 and 100, the share of zeros at which each measure switches to the forecast 0, by bisection. Analysis: For each setting and measure: the target, its quantile level and the expected score there; for each setting, the relative difference of the msMAPE target from the sMAPE target; and the two zero thresholds. Reported as tables, with no decision rule.

msMAPE beside sMAPE: near zero, the two targets as a share of the median against the share of zeros, for lognormal outcomes at medians 1 and 100, with the zero…
A53 msMAPE beside sMAPE: near zero, the two targets as a share of the median against the share of zeros, for lognormal outcomes at medians 1 and 100, with the zero thresholds at median 1; away from zero, the relative difference of the msMAPE target from the sMAPE target for each distribution; on Poisson counts, the two targets against the mean. Drawn by lab run RN11 from the recorded results of RN10 (lab/runs/RN11/figures/msmape_targets.png).
E16.T1 · The msMAPE target beside the sMAPE target, with their quantile levels and the relative difference of the msMAPE target from the sMAPE target
Rowsettingtarget (measure: sMAPE)target (measure: msMAPE)quantile level of the target (measure: sMAPE)quantile level of the target (measure: msMAPE)relative difference of the msMAPE target from the sMAPE targetshare of zero outcomes
r1lognormal sigma=0.51001000.50.50.00000
r2lognormal sigma=110099.980.50.4999-0.00020
r3inverse Gaussian shape=1014.0213.980.43870.438-0.00270
r4inverse Gaussian shape=10066.5366.520.49290.4929-0.00010
r5log-skewed left sd=1 skew=-1128.3128.20.53620.5361-0.00020
r6log-skewed right sd=1 skew=+177.9677.960.46380.46380.00000
r7lognormal median=1 sigma=110.98460.50.4938-0.01540
r8lognormal median=1 sigma=1 with zeros p0=0.210.90540.60.5683-0.09460.2
r9lognormal median=1 sigma=1 with zeros p0=0.400.17020.40.4230.4
r10lognormal median=1 sigma=1 with zeros p0=0.6000.60.600.6
r11lognormal median=1 sigma=1 with zeros p0=0.8000.80.800.8
r12lognormal median=100 sigma=110099.980.50.4999-0.00020
r13lognormal median=100 sigma=1 with zeros p0=0.210099.890.60.5997-0.00110.2
r14lognormal median=100 sigma=1 with zeros p0=0.4000.40.400.4
r15lognormal median=100 sigma=1 with zeros p0=0.6000.60.600.6
r16lognormal median=100 sigma=1 with zeros p0=0.8000.80.800.8
r17Poisson mean=0.3000.74080.740800.7408
r18Poisson mean=0.5000.60650.606500.6065
r19Poisson mean=1110.73580.735800.3679
r20Poisson mean=1.5220.80880.808800.2231
r21Poisson mean=3330.64720.647200.0498
r22Poisson mean=1010100.5830.58300.0000
E16.T2 · The share of zeros at which each measure's target switches to the forecast 0, for lognormal outcomes with sigma 1
Rowsettingzero threshold (measure: sMAPE)zero threshold (measure: msMAPE)
r1lognormal median=1 sigma=10.3940.47
r2lognormal median=100 sigma=10.3940.3937

E17 · quant · other · cited by C17

Numerical checks of the formulas of C17 (formula checks), lab run RN9 under the formal plan that decided it: parts (a), (b) and (c) on 200000 random pairs of any sign and scale, with exact zeros and pairs near the switch at |y|+|f| = 0.5; every check is within the plan's tolerances.

E17.T1 · maximum absolute difference between s~ and s (|y|+|f|)/(|y|+|f|+delta) where |y|+|f| >= 0.5, violations of 5/6 <= (|y|+|f|)/(|y|+|f|+delta) <= 1 where |y|+|f| >= 0.5, maximum absolute difference between s~ and |y-f|/(0.5+delta) where |y|+|f| < 0.5, violations of 0 <= m < 1, msMAPE term of a zero forecast of a zero actual, maximum absolute difference between s~(0, f) and |f|/(0.5+delta) for 0 < |f| < 0.5, pairs with |y|+|f| >= 0.5, pairs with |y|+|f| < 0.5; pairs: 200000, seed: 7 (lab runs RN9) (pairs: 200000, seed: 7)
Rowmaximum absolute difference between s~ and s (|y|+|f|)/(|y|+|f|+delta) where |y|+|f| >= 0.5violations of 5/6 <= (|y|+|f|)/(|y|+|f|+delta) <= 1 where |y|+|f| >= 0.5maximum absolute difference between s~ and |y-f|/(0.5+delta) where |y|+|f| < 0.5violations of 0 <= m < 1msMAPE term of a zero forecast of a zero actualmaximum absolute difference between s~(0, f) and |f|/(0.5+delta) for 0 < |f| < 0.5pairs with |y|+|f| >= 0.5pairs with |y|+|f| < 0.5
r10.00000000013288467116

E18 · formal · cited by C1

Proof of C1 (section H1 of the proof document A1, rendered in A2). Only the ratio of the actual to the forecast matters, not their size.

  1. Since yy and ff are positive, |y|+|f|=y+f|y| + |f| = y + f. So s=|y−f|/(y+f)s = |y - f|/(y + f). (algebra; from D1)
  2. Dividing the numerator and the denominator by f>0f > 0 gives |y−f|/f(y+f)/f\dfrac{|y-f|/f}{(y+f)/f}. Since f>0f > 0, |y−f|/f=|(y−f)/f|=|r−1||y - f|/f = |(y - f)/f| = |r - 1|. And (y+f)/f=r+1(y+f)/f = r + 1. (algebra; from S1)
  3. So s=|r−1|/(r+1)s = |r - 1|/(r + 1). The contribution depends only on the ratio of actual to forecast. (conclusion; from S2)

E19 · formal · cited by C2

Proof of C2 (section H2 of the proof document A1, rendered in A2). sMAPE is symmetric in ratios. It favours a high forecast only when errors are compared in absolute terms.

  1. By C1, s=|r−1|/(r+1)s = |r - 1|/(r + 1). Replacing rr by 1/r1/r gives |1/r−1|1/r+1\dfrac{|1/r - 1|}{1/r + 1}. Multiplying the numerator and the denominator by r>0r > 0 gives |1−r|1+r\dfrac{|1 - r|}{1 + r}, the same value. (algebra; from C1)
  2. The forecast f=kyf = ky has r=1/kr = 1/k, and f=y/kf = y/k has r=kr = k. By step 1 they score the same. This is (a). (algebra; from S1)
  3. The forecasts y+dy + d and y−dy - d are both dd away from yy, so both numerators are dd. The denominators are y+(y+d)=2y+dy + (y + d) = 2y + d and y+(y−d)=2y−d>0y + (y - d) = 2y - d > 0. The larger denominator gives the smaller value. This is (b). (algebra; from S2)
  4. So sMAPE treats a forecast kk times too high and one kk times too low alike. It favours the high forecast only when the two are equally wrong in absolute terms. (conclusion; from S3)

E20 · formal · cited by C3

Proof of C3 (section H3 of the proof document A1, rendered in A2). sMAPE measures the error on the log scale. For small errors it is half the absolute log error; for large errors it levels off.

  1. Write z=ln⁡y−ln⁡fz = \ln y - \ln f, so that r=y/f=ezr = y/f = e^z. By C1, s=|ez−1|ez+1s = \dfrac{|e^z - 1|}{e^z + 1}. (definition; from C1)
  2. Multiplying the numerator and the denominator of ez−1ez+1\dfrac{e^z - 1}{e^z + 1} by e−z/2e^{-z/2} gives ez/2−e−z/2ez/2+e−z/2=tanh⁡(z/2)\dfrac{e^{z/2} - e^{-z/2}}{e^{z/2} + e^{-z/2}} = \tanh(z/2). The denominator is positive, so s=|tanh⁡(z/2)|s = |\tanh(z/2)|. (algebra; from S1)
  3. tanh⁡\tanh changes only its sign when its argument changes sign: tanh⁡(−x)=−tanh⁡(x)\tanh(-x) = -\tanh(x), as the definition shows. So |tanh⁡(z/2)|=tanh⁡(|z|/2)|\tanh(z/2)| = \tanh(|z|/2), which is the stated form. (algebra; from S2)
  4. For (a): tanh⁡\tanh is 0 at 0. On [0,∞)[0, \infty) it is strictly increasing and concave, and it approaches 1 as its argument grows; its derivative 1−tanh2⁡x1 - \tanh^2 x is positive and decreasing for x>0x > 0. With the argument |ln⁡y−ln⁡f|/2|\ln y - \ln f|/2, ss has these properties in |ln⁡y−ln⁡f||\ln y - \ln f|. (monotonicity; from S3)
  5. For (b): for every x≥0x \ge 0, x−x3/3≤tanh⁡x≤xx - x^3/3 \le \tanh x \le x. Putting x=|ln⁡y−ln⁡f|/2x = |\ln y - \ln f|/2 gives the bounds. (algebra; from S4)
  6. The mean of |ln⁡yt−ln⁡f||\ln y_t - \ln f| over a test set is minimised when ln⁡f\ln f is a median of the ln⁡yt\ln y_t. Since the logarithm is increasing, this is when ff is a median of the yty_t. (monotonicity; from S5)
  7. So s=tanh⁡(|ln⁡y−ln⁡f|/2)s = \tanh(|\ln y - \ln f|/2). It rises from 0 like half the absolute log error and levels off at 1. (conclusion; from S6)

E21 · formal · cited by C9

Proof of C9 (section H4 of the proof document A1, rendered in A2). The expected values of RMSE, MAE and MAPE become infinite for every forecast when the actuals have an infinite moment, such as an infinite mean (a heavy tail, C8). The expected sMAPE never does, and there is always a best forecast.

  1. Finite. For positive values, s=tanh⁡(|ln⁡y−ln⁡f|/2)s = \tanh(|\ln y - \ln f|/2) (C3), which lies between 0 and 1. For any signs, |y−f|≤|y|+|f||y - f| \le |y| + |f|, so again s≤1s \le 1. A quantity between 0 and 1 has an expectation between 0 and 1, so the expected sMAPE is between 0 and 200, whatever the actuals. (earlier result; from C3)
  2. A best forecast. Write E[s(y,f)]\mathrm E[s(y,f)] for the expectation over the random actual yy. As ff approaches 0 or grows without bound, |ln⁡y−ln⁡f||\ln y - \ln f| grows for every positive value of yy, so ss approaches 1 (C3), and so does its expectation. For every positive ff it is below 1. It changes continuously with ff, because ss does and is at most 1 (bounded convergence). A continuous function that is below 1 in between and returns towards 1 at both ends reaches a lowest value at some f>0f > 0. A forecast of 0 or a negative forecast scores 1 on every positive actual, so it is never better. (bounded convergence; from C3, D1)
  3. So the expected sMAPE is always finite, whatever the tails of the actuals. For positive actuals, a forecast with the lowest expected sMAPE always exists. (conclusion; from S2)

E22 · formal · cited by C4

Proof of C4 (section H5 of the proof document A1, rendered in A2). For any distribution, the best forecast balances the actuals below it and above it. Each actual is weighted by how close it is to the forecast by ratio. This also says where the target lies relative to the median.

  1. The derivative of ss. When y>f>0y > f > 0, s=(y−f)/(y+f)s = (y - f)/(y + f). By the quotient rule its derivative in ff is −(y+f)−(y−f)(y+f)2=−2w(y,f)\dfrac{-(y+f) - (y-f)}{(y+f)^2} = -2w(y,f). When f>y>0f > y > 0, s=(f−y)/(y+f)s = (f - y)/(y + f), and in the same way its derivative is 2w(y,f)2w(y,f). Together: for y≠fy \ne f, the derivative is 2w(y,f)(𝟏{y<f}−𝟏{y>f})2w(y,f)(\mathbf 1\{y < f\} - \mathbf 1\{y > f\}). (quotient rule; from D1)
  2. A bound on it. Since (y+f)2≥4yf(y+f)^2 \ge 4yf, w(y,f)≤1/(4f)w(y,f) \le 1/(4f). So the derivative of ss is at most 1/(2f)1/(2f) in size, whatever yy. The only actual at which ss has no derivative is y=fy = f, which has probability zero. (algebra; from S1)
  3. The derivative of the expectation. With this bound, the derivative may be taken inside the expectation (dominated convergence): ddfE[s(y,f)]=2E[w(y,f)(𝟏{y<f}−𝟏{y>f})].\frac{d}{df}\,\mathrm E[s(y,f)] = 2\,\mathrm E\big[w(y,f)\,(\mathbf 1\{y < f\} - \mathbf 1\{y > f\})\big]. (dominated convergence; from S2)
  4. The balance. A forecast with the lowest expected sMAPE exists (C9). There this derivative is 0, which is the stated balance: half the total weight lies below ff and half above. (earlier result; from C9)
  5. The weight. With r=y/fr = y/f, w(y,f)=1f⋅r(1+r)2=1f⋅1r+2+1/rw(y,f) = \dfrac1f\cdot\dfrac{r}{(1+r)^2} = \dfrac1f\cdot\dfrac{1}{r + 2 + 1/r}. The sum r+1/rr + 1/r is the same for rr and 1/r1/r. It is smallest, 2, at r=1r = 1, and grows as rr moves away from 1. So ww is largest at y=fy = f and falls as y/fy/f moves away from 1, equally for rr and 1/r1/r. (algebra; from S4)
  6. The shape of expected sMAPE. With a single local minimum, the expected sMAPE decreases up to it and increases after it. Otherwise it would turn again, at a local maximum. After that turn it would have to fall and then rise once more, because it returns towards 1 at both ends (C9). That would give a second local minimum. (earlier result; from C9)
  7. Where the target lies. At f=mf = m, by step 3 the derivative is twice the weight below mm minus the weight above mm. If the actuals above carry more weight, the derivative is negative: the expected sMAPE is decreasing at the median. By step 6, the median then lies before the minimum, so the target is above the median. The case where the actuals below carry more weight is the mirror image. (symmetry; from S3, S6)
  8. The second derivative. Measure the forecast on the log scale, c=ln⁡fc = \ln f, so that the expected contribution is E[tanh⁡(|ln⁡y−c|/2)]\mathrm E[\tanh(|\ln y - c|/2)] (C3). The derivative of tanh⁡(|x−c|/2)\tanh(|x - c|/2) in cc is −12sech2⁡(x−c2)-\tfrac12\operatorname{sech}^2\big(\tfrac{x-c}{2}\big) for c<xc < x and +12sech2⁡(x−c2)+\tfrac12\operatorname{sech}^2\big(\tfrac{x-c}{2}\big) for c>xc > x: it jumps by 1 at c=xc = x. Differentiating again and taking the expectation over x=ln⁡yx = \ln y, the jumps add the density gg of ln⁡y\ln y at cc, and elsewhere the second derivative is −12sech2⁡(x−c2)tanh⁡(|x−c|2)-\tfrac12\operatorname{sech}^2\big(\tfrac{x-c}{2}\big)\tanh\big(\tfrac{|x-c|}{2}\big). So the second derivative of the expected contribution is g(c)−12E[sech2⁡(ln⁡y−c2)tanh⁡(|ln⁡y−c|2)].g(c) - \tfrac12\,\mathrm E\Big[\operatorname{sech}^2\Big(\frac{\ln y - c}{2}\Big)\tanh\Big(\frac{|\ln y - c|}{2}\Big)\Big]. (earlier result; from C3)
  9. The sufficient condition. If the two weights are equal, the first derivative at the median is 0 (step 3). If, in addition, the stated inequality holds, the second derivative at c=ln⁡mc = \ln m is positive (step 8), so the median is a strict local minimum. With a single local minimum, it is the target. (algebra; from S3, S8)
  10. Without the inequality, a point where the first derivative is 0 can be a flat point of inflection, where the expected sMAPE keeps falling on one side and rising on the other. The independent checker built such a case: a mixture whose only minimum lies far below the median. So equal weights alone do not make the median the target. (logical argument; from S9)
  11. So sMAPE targets a weighted median of the actuals. Each actual is weighted by how close it is to the forecast by ratio. The target is above the median when the actuals above the median carry more of this weight, and below it when those below carry more. When the two balance, the median is the target if, in addition, enough actuals lie near it, relative to the pull of those further away, as the stated inequality says. (conclusion; from S10)

E23 · formal · cited by C5

Proof of C5 (section H6 of the proof document A1, rendered in A2). This is the case of C4 in which the weights balance at the median.

  1. By C3, E[s(y,f)]=E[tanh⁡(|ln⁡y−ln⁡f|/2)]\mathrm E[s(y,f)] = \mathrm E\big[\tanh\big(|\ln y - \ln f|/2\big)\big]. Take a forecast f=meτf = m\,e^{\tau} with τ>0\tau > 0; the case τ<0\tau < 0 is the mirror image. We show that its expected sMAPE is larger than that of mm. (setup; from C3)
  2. Write both expectations as integrals over vv, the distance of ln⁡y\ln y from ln⁡f\ln f. For the forecast meτm\,e^{\tau}, ln⁡y=ln⁡m+τ+v\ln y = \ln m + \tau + v; for mm, ln⁡y=ln⁡m+v\ln y = \ln m + v. Their difference is Δ=∫tanh⁡(|v|/2)[g(ln⁡m+τ+v)−g(ln⁡m+v)]dv.\Delta = \int \tanh(|v|/2)\,\big[g(\ln m+\tau+v) - g(\ln m+v)\big]\,dv. (definition; from S1)
  3. Write ϕ(v)=g(ln⁡m+τ+v)−g(ln⁡m+v)\phi(v) = g(\ln m+\tau+v) - g(\ln m+v). By the symmetry of gg about ln⁡m\ln m, ϕ(−τ−v)=g(ln⁡m−v)−g(ln⁡m−τ−v)=g(ln⁡m+v)−g(ln⁡m+τ+v)=−ϕ(v)\phi(-\tau-v) = g(\ln m-v) - g(\ln m-\tau-v) = g(\ln m+v) - g(\ln m+\tau+v) = -\phi(v). So reflecting vv about −τ/2-\tau/2 reverses the sign of ϕ\phi. (definition; from S2)
  4. Split the integral at −τ/2-\tau/2 and reflect its left part onto the right. This gives Δ=∫v>−τ/2[tanh⁡(|v|/2)−tanh⁡(|v+τ|/2)]ϕ(v)dv.\Delta = \int_{v > -\tau/2} \big[\tanh(|v|/2) - \tanh(|v+\tau|/2)\big]\,\phi(v)\,dv. (algebra; from S3)
  5. For v>−τ/2v > -\tau/2, vv is closer to 0 than v+τv + \tau is: |v|<|v+τ||v| < |v + \tau|. So the bracket is negative, because tanh⁡\tanh is increasing (C3). And ϕ(v)≤0\phi(v) \le 0 by unimodality: g(ln⁡m+τ+v)≤g(ln⁡m+v)g(\ln m+\tau+v) \le g(\ln m+v). Their product is ≥0\ge 0, and >0> 0 on a set of positive probability. So Δ>0\Delta > 0. (monotonicity; from C3)
  6. So every forecast other than the median has a larger expected sMAPE. The median is the only forecast that minimises it. (conclusion; from S5)

E24 · formal · cited by C6

Proof of C6 (section H7 of the proof document A1, rendered in A2). When the actuals form two separate groups, the median falls in the gap between them, far from both. So sMAPE scores it worse than forecasts near either group.

  1. Write forecasts on the log scale as ln⁡f=ln⁡m+c\ln f = \ln m + c. By C3, the expected contribution is E[tanh⁡(|ln⁡y−ln⁡m−c|/2)]\mathrm E[\tanh(|\ln y - \ln m - c|/2)], a function of cc. (setup; from C3)
  2. Why the median is bad. Take the groups to be single points first. A forecast at the median is aa away, on the log scale, from every actual. So its expected ss is tanh⁡(a/2)\tanh(a/2). (case analysis; from S1)
  3. A forecast at the mode, ln⁡f=ln⁡m+a\ln f = \ln m + a, is exact for half the actuals and 2a2a away from the other half. So its expected ss is 12tanh⁡(a)\tfrac12\tanh(a). Since tanh⁡(a)=2tanh⁡(a/2)1+tanh2⁡(a/2)\tanh(a) = \dfrac{2\tanh(a/2)}{1 + \tanh^2(a/2)}, this is tanh⁡(a/2)1+tanh2⁡(a/2)<tanh⁡(a/2)\dfrac{\tanh(a/2)}{1 + \tanh^2(a/2)} < \tanh(a/2). The mode is better. (algebra; from S2)
  4. The expected ss changes continuously as the groups widen from points, because ss is bounded and continuous. So the mode is still better for narrow groups. (continuity; from S3)
  5. Why the median is a local maximum. By C4, step 8, the second derivative of the expected contribution at the median is g(ln⁡m)−12E[sech2⁡(ln⁡y−ln⁡m2)tanh⁡(|ln⁡y−ln⁡m|2)].g(\ln m) - \tfrac12\,\mathrm E\Big[\operatorname{sech}^2\Big(\frac{\ln y - \ln m}{2}\Big)\tanh\Big(\frac{|\ln y - \ln m|}{2}\Big)\Big]. (earlier result; from C4)
  6. By symmetry the first derivative at the median is 0. When the groups narrow, few actuals lie near the median, so g(ln⁡m)g(\ln m) tends to 0. Meanwhile the expectation tends to sech2⁡(a/2)tanh⁡(a/2)>0\operatorname{sech}^2(a/2)\tanh(a/2) > 0. So the second derivative becomes negative, and the median is a local maximum. (symmetry; from S5)
  7. A minimiser on each side. The expected sMAPE has a lowest point (C9). It is not at the local maximum c=0c = 0. By symmetry, if c*c^* is a minimiser, so is −c*-c^*. (symmetry; from C9)
  8. Near the modes. Write tanh⁡(|z|/2)=1−2λ(z)\tanh(|z|/2) = 1 - 2\lambda(z), with λ(z)=1/(1+e|z|)\lambda(z) = 1/(1 + e^{|z|}). The function λ\lambda is largest at 0 and falls as |z||z| grows, more slowly the larger |z||z| is. Minimising the expected ss is maximising E[λ(ln⁡y−ln⁡m−c)]\mathrm E[\lambda(\ln y - \ln m - c)]. As the groups narrow, this approaches pλ(c+a)+(1−p)λ(c−a)p\,\lambda(c + a) + (1 - p)\,\lambda(c - a), uniformly in cc. (definition; from S7)
  9. With equal weights, take 0<c<a0 < c < a. The derivative of this limit is half the rate at which λ\lambda falls at distance a−ca - c, minus half the rate at distance a+ca + c. It is positive, because λ\lambda falls faster nearer 0. For c>ac > a the limit decreases. So on c>0c > 0 its only maximum is at c=ac = a, and the minimisers on that side approach aa. By symmetry, those on the other side approach −a-a. This proves (a). (symmetry; from S8)
  10. Unequal weights. The limit increases up to c=−ac = -a and decreases after c=ac = a. In between it is convex, by the argument of step 9. So its maximum is at −a-a or at aa. The difference between these two values is (2p−1)(λ(0)−λ(2a))(2p - 1)\big(\lambda(0) - \lambda(2a)\big), which is positive when p>1/2p > 1/2. So the maximum, and with it the best forecast for narrow groups, is at the mode of the larger group. This proves (b). (convexity; from S9)
  11. So with two separate groups of actuals, the median is a hump of the expected sMAPE. The best forecasts are near the modes, and with unequal groups the larger one wins. As the weights pass through equality, the target jumps from one mode to the other. (conclusion; from S10)

E25 · formal · cited by C7

Proof of C7 (section H8 of the proof document A1, rendered in A2). The worst possible score goes to forecasts of the wrong sign and to zero forecasts. With enough zeros among the actuals, sMAPE prefers the forecast 0. The share of zeros needed is below one half.

  1. When yy and ff have opposite signs, |y−f|=|y|+|f||y - f| = |y| + |f|, so s=1s = 1. When exactly one of them is 0, the numerator and the denominator both equal the absolute value of the other one, so s=1s = 1. (algebra; from D1)
  2. A positive forecast scores 1 on every zero actual (step 1). So its expected ss is p0p_0 plus (1−p0)(1 - p_0) times its expected ss on the positive actuals. This is at least p0+(1−p0)s+p_0 + (1-p_0)\,s_+, with equality at the best positive forecasts, which exist (C9). (earlier result; from S1, C9)
  3. The forecast 0 scores 0 on zero actuals and 1 on positive ones. So its expected ss is 1−p01 - p_0. A negative forecast scores 1 on every actual, which is never better than 0. (algebra; from S2)
  4. The forecast 0 is better when 1−p0<p0+(1−p0)s+1 - p_0 < p_0 + (1 - p_0)s_+. Collecting the terms in p0p_0 gives 1−s+<p0(2−s+)1 - s_+ < p_0(2 - s_+). Since 2−s+>02 - s_+ > 0, this is the stated inequality. (algebra; from S3)
  5. Compare the threshold with 1/21/2: 1−s+2−s+<12\dfrac{1 - s_+}{2 - s_+} < \dfrac12 exactly when 2−2s+<2−s+2 - 2s_+ < 2 - s_+, that is, when s+>0s_+ > 0. And s+>0s_+ > 0 unless the positive actuals are all one value. (algebra; from S4)
  6. So sMAPE switches to the forecast 0 once zeros are frequent enough, and the share needed is below one half. At the threshold, the target jumps from a positive forecast to 0. (conclusion; from S5)

E26 · formal · cited by C11

Proof of C11 (section H9 of the proof document A1, rendered in A2). When actuals can be negative or positive, sMAPE's target is never near zero. It is on the same side of zero as the mean. So a small shift of the mean across zero makes the target jump from a negative value to a positive one.

  1. A positive forecast against its mirror image. Take f>0f > 0 and compare it with −f-f. An actual y>0y > 0 has the sign of ff, so it scores less than 1 with ff, and exactly 1 with −f-f (C7). (symmetry; from C7)
  2. The mirror actual −y-y scores 1 with ff. By symmetry, it scores the same amount less than 1 with −f-f. So each pair of actuals yy and −y-y favours ff when yy is the more likely of the two, and −f-f otherwise. (symmetry; from S1)
  3. When μ>0\mu > 0, every positive actual yy is more likely than −y-y, because yy is closer to μ\mu and the density of the actuals falls away from μ\mu. So every pair favours ff, and ff has the lower expected sMAPE. Every best forecast is therefore positive. The case μ<0\mu < 0 is the mirror image. (symmetry; from S2)
  4. Best forecasts exist on each side. For a forecast f>0f > 0, the actuals of its sign score below 1 and have positive probability, so its expected ss is below 1. As ff approaches 0 or grows without bound, every non-zero actual scores close to 1, so the expected ss approaches 1. As in C9, the expected ss is continuous in ff and so reaches a lowest value for f>0f > 0; the same holds for f<0f < 0. (continuity; from C9)
  5. Away from zero, uniformly. Fix a forecast f1>0f_1 > 0. For every μ\mu in a range around 0, say |μ|≤1|\mu| \le 1, its expected ss is at most some value q<1q < 1, since it changes continuously with μ\mu. For a forecast ff with |f|≤η|f| \le \eta, every actual with |y|≥η|y| \ge \sqrt\eta scores at least (η−η)/(η+η)(\sqrt\eta - \eta)/(\sqrt\eta + \eta), which approaches 1 as η\eta shrinks; and the share of actuals with |y|<η|y| < \sqrt\eta is small for every such μ\mu: since ε\varepsilon is symmetric and its density decreases away from 0, an interval of a given length around 0 holds at least as much of its probability as any other interval of that length, so P(|μ+ε|<η)≤P(|ε|<η)P(|\mu + \varepsilon| < \sqrt\eta) \le P(|\varepsilon| < \sqrt\eta), which tends to 0 as η\eta shrinks. So for η\eta small enough, every forecast within η\eta of 0 has an expected ss above qq, for every μ\mu in the range: it is worse than f1f_1 (or its mirror image). The best forecasts therefore stay at least η\eta from 0. (symmetry; from S4)
  6. So the target is always on the same side of zero as the mean, and never near zero. It jumps from negative to positive as the mean crosses zero. (conclusion; from S5)

E27 · formal · cited by C8

Proof of C8 (section H10 of the proof document A1, rendered in A2). Each of the other three measures has an infinite expected value, for every forecast, when a moment it needs is infinite. What it then rewards depends on the measure.

  1. (a) RMSE. If E[y2]\mathrm E[y^2] is finite, so is E[y]\mathrm E[y]. Expanding the square gives E[(y−f)2]=E[y2]−2fE[y]+f2\mathrm E[(y-f)^2] = \mathrm E[y^2] - 2f\,\mathrm E[y] + f^2, which is finite. Conversely, y2≤2(y−f)2+2f2y^2 \le 2(y - f)^2 + 2f^2. So a finite E[(y−f)2]\mathrm E[(y-f)^2] makes E[y2]\mathrm E[y^2] finite. (algebra; from D10)
  2. Expanding both squares gives (y−f1)2−(y−f2)2=(f2−f1)(2y−f1−f2)(y - f_1)^2 - (y - f_2)^2 = (f_2 - f_1)(2y - f_1 - f_2). Its expectation is finite when E[y]\mathrm E[y] is, and equals (f1−f2)(f1+f2−2E[y])=(f1−E[y])2−(f2−E[y])2(f_1 - f_2)(f_1 + f_2 - 2\,\mathrm E[y]) = (f_1 - \mathrm E[y])^2 - (f_2 - \mathrm E[y])^2. (algebra; from S1)
  3. For f1=E[y]f_1 = \mathrm E[y] this is negative for every f2≠E[y]f_2 \ne \mathrm E[y]. So the mean is preferred. (algebra; from S2)
  4. If E[y]\mathrm E[y] is infinite, then for f2>f1f_2 > f_1 the expectation in step 2 is +∞+\infty. Every forecast is beaten by any larger one, so none is preferred to all others. (algebra; from S2)
  5. (b) MAE. Since |y−f|≥y−f|y - f| \ge y - f, E|y−f|\mathrm E|y - f| is infinite when E[y]\mathrm E[y] is. Otherwise it is at most E[y]+|f|\mathrm E[y] + |f|. (algebra; from D11)
  6. The difference |y−f1|−|y−f2||y - f_1| - |y - f_2| is at most |f1−f2||f_1 - f_2| in size, so its expectation is finite. (algebra; from S5)
  7. The derivative of |y−f||y - f| in ff is 𝟏{y<f}−𝟏{y>f}\mathbf 1\{y < f\} - \mathbf 1\{y > f\}. The expectation of this is P(y<f)−P(y>f)P(y < f) - P(y > f), which is negative below a median and positive above it. So a median is preferred. (algebra; from S6)
  8. (c) MAPE. For f>0f > 0, the contribution of an actual to MAPE, divided by 100, is |y−f|/y=|1−f/y||y - f|/y = |1 - f/y|. This is at least f/y−1f/y - 1. So the expected MAPE is infinite when E[1/y]\mathrm E[1/y] is, and at most 1+fE[1/y]1 + f\,\mathrm E[1/y] otherwise. (algebra; from D12)
  9. As a function of ff, |1−f/y||1 - f/y| is convex. Its derivative is (1/y)(𝟏{y<f}−𝟏{y>f})(1/y)(\mathbf 1\{y < f\} - \mathbf 1\{y > f\}). Its expectation is 0 where the actuals below ff and those above it have equal total weight 1/y1/y: at a median of the actuals weighted by 1/y1/y. (convexity; from S8)
  10. The forecast 0 gives |1−0|=1|1 - 0| = 1 for every actual, so MAPE 100. A negative forecast gives more than 1. So when E[1/y]\mathrm E[1/y] is infinite, 0 is the only forecast with a finite expected MAPE. (algebra; from S9)
  11. If y=0y = 0 with positive probability: for f≠0f \ne 0 that actual contributes |f|/0|f|/0, which is infinite. For f=0f = 0 it contributes 0/00/0, which is undefined. (algebra; from S10)
  12. So each of RMSE, MAE and MAPE has an infinite expected value when a moment it needs is infinite. RMSE and MAE still rank forecasts by their differences. MAPE turns to the forecast 0, and with zero actuals it targets nothing. (conclusion; from S11)

E28 · formal · cited by C10

Proof of C10 (section H11 of the proof document A1, rendered in A2). For widely spread actuals, every forecast scores close to the worst value. The differences between forecasts shrink, and the target moves from the median to the mode of ln⁡y\ln y.

  1. Rewrite the expectation. By C3, and since tanh⁡(|x|/2)=1−21+e|x|\tanh(|x|/2) = 1 - \dfrac{2}{1 + e^{|x|}}, E[s(y,f)]=1−E[2/(1+e|ln⁡y−ln⁡f|)]\mathrm E[s(y,f)] = 1 - \mathrm E\big[2/(1 + e^{|\ln y - \ln f|})\big]. The density of ln⁡y\ln y at xx is h((x−μ)/σ)/σh\big((x - \mu)/\sigma\big)/\sigma. Substituting x=ln⁡f+vx = \ln f + v in the integral gives E[s(y,f)]=1−1σ∫21+e|v|h(c+vσ)dv.\mathrm E[s(y,f)] = 1 - \frac1\sigma\int \frac{2}{1 + e^{|v|}}\,h\Big(c + \frac v\sigma\Big)\,dv. (change of variables; from C3)
  2. Expand hh. By Taylor's theorem, h(c+v/σ)=h(c)+h′(c)v/σ+12h′′(ξ)v2/σ2h(c + v/\sigma) = h(c) + h'(c)\,v/\sigma + \tfrac12 h''(\xi)\,v^2/\sigma^2 for some ξ\xi between cc and c+v/σc + v/\sigma. (Taylor's theorem; from S1)
  3. The first term integrates to 4ln⁡2h(c)4\ln 2\,h(c), since ∫21+e|v|dv=4ln⁡2\int \frac{2}{1+e^{|v|}}\,dv = 4\ln 2. Over v>0v > 0, ∫2/(1+ev)dv=2ln⁡2\int 2/(1 + e^v)\,dv = 2\ln 2. (algebra; from S2)
  4. The second term integrates to 0: its factor vv changes sign with vv, while the rest does not. (algebra; from S3)
  5. The third term is at most 12max⁡|h′′|\tfrac12\max|h''| times ∫2v21+e|v|dv=6ζ(3)≈7.21\int \frac{2v^2}{1+e^{|v|}}\,dv = 6\zeta(3) \approx 7.21, divided by σ2\sigma^2. This integral is a standard Fermi–Dirac integral. With the factor 1/σ1/\sigma of step 1, this gives the expansion. (standard integral; from S1)
  6. (a). The expansion tends to 1, so the expected sMAPE tends to 200. Two forecasts at fixed quantile levels have fixed positions c1c_1 and c2c_2. So their difference is 4ln⁡2[h(c2)−h(c1)]/σ+O(σ−3)4\ln 2\,[h(c_2) - h(c_1)]/\sigma + O(\sigma^{-3}). (algebra; from S5)
  7. (b). Minimising the expansion over cc is maximising h(c)h(c), up to a term that shrinks like σ−2\sigma^{-2} for every cc at once. A density with a bounded derivative tends to 0 far out. So its single highest point c0c_0 stands clear of every other point, and the minimisers' positions tend to c0c_0. So do their quantile levels. (algebra; from S6)
  8. (c). By C3 (b), 12σ|ε−c|−s\tfrac12\sigma|\varepsilon - c| - s lies between 0 and 124σ3|ε−c|3\tfrac1{24}\sigma^3|\varepsilon - c|^3. So for cc in any bounded range, E[s]/σ=12E|ε−c|+O(σ2)\mathrm E[s]/\sigma = \tfrac12\mathrm E|\varepsilon - c| + O(\sigma^2). This uses E|ε|3<∞\mathrm E|\varepsilon|^3 < \infty. And E|ε−c|\mathrm E|\varepsilon - c| is smallest at the median of ε\varepsilon. (earlier result; from C3)
  9. The minimisers also stay in a bounded range. For large |c||c|, most actuals have ε\varepsilon at least |c|/2|c|/2 from cc. And tanh⁡x≥xtanh⁡1\tanh x \ge x\tanh 1 for 0≤x≤10 \le x \le 1. So E[s]/σ\mathrm E[s]/\sigma grows with |c||c|, or exceeds a multiple of 1/σ1/\sigma, beyond its value at the median. So the minimisers' positions tend to the median of ε\varepsilon, whose quantile level is 1/21/2. (algebra; from S8)
  10. So as the actuals spread out, the expected sMAPE flattens towards its bound. Differences between forecasts vanish like 1/σ1/\sigma. The target moves from the median to the mode of ln⁡y\ln y. (conclusion; from S9)

E29 · formal · cited by C12

Proof of C12 from the proofs of C5 and C4 (sections H6 and H5 of the proof document A1): the two cases in which the median of the actuals is the sMAPE target.

  1. Case (a): when ln⁡y\ln y has a density symmetric about ln⁡m\ln m and unimodal, the median mm is the only forecast with the lowest expected sMAPE. (earlier result; from C5)
  2. Case (b): for positive yy that takes no single value with positive probability, the derivative of E[s(y,f)]\mathrm E[s(y,f)] in ff is 2E[w(y,f)(𝟏{y<f}−𝟏{y>f})]2\,\mathrm E[w(y,f)(\mathbf 1\{y<f\} - \mathbf 1\{y>f\})] with w(y,f)=y/(y+f)2w(y,f) = y/(y+f)^2 (C4, its proof's step 3). (earlier result; from C4)
  3. When the actuals above and below mm carry equal total weight with the forecast at mm, this derivative is 0 at f=mf = m. (substitution; from S2)
  4. With c=ln⁡fc = \ln f, the second derivative of the expected contribution at c=ln⁡mc = \ln m is g(ln⁡m)−12E[sech2⁡((ln⁡y−ln⁡m)/2)tanh⁡(|ln⁡y−ln⁡m|/2)]g(\ln m) - \tfrac12\mathrm E[\operatorname{sech}^2((\ln y - \ln m)/2)\tanh(|\ln y - \ln m|/2)] (C4, its proof's step 8). The stated inequality makes it positive, so mm is a strict local minimum. (second-derivative test; from C4, S3)
  5. A forecast with the lowest expected sMAPE exists (C9). When expected sMAPE has a single local minimum, that minimum is the lowest value, so it is at mm. (logical argument; from C9, S4)
  6. In both cases the median mm is the sMAPE target. (conclusion; from S1, S5)

E30 · formal · cited by C17

Proof of C17 (the proof document A5). msMAPE behaves like sMAPE except near zero, where it becomes an absolute error.

  1. Which branch of the maximum. ℓ+δ≥0.5+δ\ell + \delta \ge 0.5 + \delta exactly when ℓ≥0.5\ell \ge 0.5. So the denominator of s~\tilde s is ℓ+δ\ell + \delta when ℓ≥0.5\ell \ge 0.5, and 0.5+δ0.5 + \delta when ℓ<0.5\ell < 0.5. (algebra; from D1)
  2. (a), the identity. When ℓ≥0.5\ell \ge 0.5, ℓ>0\ell > 0, so s=|y−f|/ℓs = |y-f|/\ell and |y−f|=sℓ|y-f| = s\,\ell. By step 1, s~=|y−f|/(ℓ+δ)=sℓ/(ℓ+δ)\tilde s = |y-f|/(\ell+\delta) = s\,\ell/(\ell+\delta). (algebra; from S1)
  3. (a), the factor. The function ℓ↦ℓ/(ℓ+δ)=1−δ/(ℓ+δ)\ell \mapsto \ell/(\ell+\delta) = 1 - \delta/(\ell+\delta) is increasing in ℓ\ell, below 1, and tends to 1 as ℓ\ell grows. At ℓ=0.5\ell = 0.5 it equals 0.5/0.6=5/60.5/0.6 = 5/6. So for ℓ≥0.5\ell \ge 0.5 it lies in [5/6,1)[5/6, 1). (monotonicity; from S2)
  4. (a), close to ss, but not a function of the ratio. By steps 2 and 3, 0≤s−s~=sδ/(ℓ+δ)≤δ/(ℓ+δ)0 \le s - \tilde s = s\,\delta/(\ell+\delta) \le \delta/(\ell+\delta), since s≤1s \le 1 (step 7). So s~\tilde s is within δ/(ℓ+δ)\delta/(\ell+\delta) of ss, which shrinks as ℓ\ell grows. But s~\tilde s depends on the scale: (y,f)=(1,2)(y,f) = (1,2) and (10,20)(10,20) have the same ratio and the same s=1/3s = 1/3, while s~=1/3.1\tilde s = 1/3.1 and 10/30.110/30.1. So s~\tilde s is not a function of y/fy/f alone. (algebra; from S2)
  5. (a), counts. If y≥1y \ge 1, then ℓ≥1≥0.5\ell \ge 1 \ge 0.5. If y=0y = 0, then ℓ=|f|\ell = |f|, which is at least 0.5 unless |f|<0.5|f| < 0.5. So on counts only a zero actual with |f|<0.5|f| < 0.5 has ℓ<0.5\ell < 0.5. (algebra; from S4)
  6. (b). When ℓ<0.5\ell < 0.5, by step 1, s~=|y−f|/(0.5+δ)\tilde s = |y-f|/(0.5+\delta). For y=f=0y = f = 0 this is 0. For y=0y = 0 and |f|<0.5|f| < 0.5 it is |f|/(0.5+δ)|f|/(0.5+\delta). And s(0,f)=|f|/|f|=1s(0,f) = |f|/|f| = 1 for f≠0f \ne 0, the largest value of ss (step 7). (algebra; from S1)
  7. (c). |y−f|≤|y|+|f|=ℓ|y - f| \le |y| + |f| = \ell (the triangle inequality), so s≤1s \le 1. When ℓ≥0.5\ell \ge 0.5, s~≤ℓ/(ℓ+δ)<1\tilde s \le \ell/(\ell+\delta) < 1 (step 3). When ℓ<0.5\ell < 0.5, s~≤ℓ/(0.5+δ)<0.5/0.6<1\tilde s \le \ell/(0.5+\delta) < 0.5/0.6 < 1. And s~≥0\tilde s \ge 0 because it is a ratio of a non-negative number to a positive one. A quantity between 0 and 1 has an expectation between 0 and 1, whatever the joint distribution of actuals and forecasts, so the expected msMAPE is between 0 and 200. (algebra; from S3)
  8. So msMAPE is the sMAPE term shrunk by a factor between 5/6 and 1 wherever |y|+|f|≥0.5|y| + |f| \ge 0.5, and an absolute error where |y|+|f|<0.5|y| + |f| < 0.5: it differs from sMAPE in kind only near zero. (conclusion; from S7)

Limitations

L1 · analytic flexibility · could bias magnitude · affects C13 · mitigation: none

Lab run RN1 deviated from plan PL1: RN1 ran at commit 3cb6c7c, not the plan's 92cd9a5. Between them only flatness.py and plot_flatness.py (plan PL3), test_smape.py and measures.py changed; in measures.py the unused function log_mae was removed (the supervisor dropped log-MAE from PL3, DC3). targets.py, theory.py, dists.py and the measures targets.py uses are unchanged. Reason: The commit of the PL3 revision came between PL1's plan and its run; it does not touch what PL1 runs.

L2 · analytic flexibility · could bias magnitude · affects C15 · mitigation: none

Lab run RN7 deviated from plan PL2: RN7 ran efficiency.py at commit fa0ce62, not the plan's 92cd9a5: the script runs its 16 settings in parallel processes (one per core) instead of one after another; theory.py also gained a function (balance_exact) that efficiency.py does not use. Each (setting, test-set size) draws from its own seed as before, and on a subset the parallel code and the plan's code gave byte-identical results.json. Reason: Three earlier attempts on one core (over an hour) were killed by container restarts before the lab recorded them (notes NT2.R2, NT5, NT6); in parallel the run finished in about 20 minutes. The analysis, the settings, the seeds and the results are unchanged.

L3 · sampling · reduces transferability · affects C14, C15 · mitigation: none

The test sets are independent draws from a fixed distribution. Real forecast errors are often correlated and their distribution can change over a test period, so a real test set carries less information than the same number of outcomes here.

L4 · model specification · reduces transferability · affects C13, C14, C15, C16 · mitigation: none

Only the distribution families studied were computed; the formal claims (C4, C10) say what holds beyond them, the simulations only illustrate it.

References

  1. Anderson, T. W., 1955. The integral of a symmetric unimodal function over a symmetric convex set and some probability inequalities. Proceedings of the American Mathematical Society 6, 170–176. https://doi.org/10.1090/S0002-9939-1955-0069229-1 Used by: Context
  2. Chen, C., Twycross, J., Garibaldi, J. M., 2017. A new accuracy measure based on bounded relative error for time series forecasting. PLOS ONE 12, e0174202. https://doi.org/10.1371/journal.pone.0174202 Used by: Context
  3. Goodwin, P., Lawton, R., 1999. On the asymmetry of the symmetric MAPE. International Journal of Forecasting 15, 405–408. https://doi.org/10.1016/S0169-2070(99)00007-2 Used by: Context
  4. Hewamalage, H., Bergmeir, C., Bandara, K., 2021. Recurrent Neural Networks for Time Series Forecasting: Current status and future directions. International Journal of Forecasting 37, 388–427. https://doi.org/10.1016/j.ijforecast.2020.06.008 Used by: Context, D28
  5. Hewamalage, H., Ackermann, K., Bergmeir, C., 2023. Forecast evaluation for data scientists: common pitfalls and best practices. Data Mining and Knowledge Discovery 37, 788–832. https://doi.org/10.1007/s10618-022-00894-5 Used by: Context
  6. Hyndman, R. J., Koehler, A. B., 2006. Another look at measures of forecast accuracy. International Journal of Forecasting 22, 679–688. https://doi.org/10.1016/j.ijforecast.2006.03.001 Used by: Context
  7. Kolassa, S., 2026. The Symmetric Mean Absolute Percentage Error: Unnecessary or Dangerous. Forecasting 8, 24. https://doi.org/10.3390/forecast8020024 Used by: Context
  8. Makridakis, S., 1993. Accuracy measures: theoretical and practical concerns. International Journal of Forecasting 9, 527–529. https://doi.org/10.1016/0169-2070(93)90079-3 Used by: D2
  9. Svetunkov, I., 2026a. Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2 Claims cited: C6, C33. Used by: Context
  10. Svetunkov, I., 2026b. Which forecast does each error measure reward? A simulation study of the functionals targeted by RMSE, MAE, MAPE and sMAPE. EntryLib. entrylib:2026.000009@v1 Claims cited: C5, C6, C11. Used by: Context
  11. Tofallis, C., 2015. A better measure of relative prediction accuracy for model selection and model estimation. Journal of the Operational Research Society 66, 1352–1362. https://doi.org/10.1057/jors.2014.103 Used by: Context

Artefacts

  • A1 · proof Proofs of the study's formal results (paper numbering H1 to H11; record H1, H2, H3, H13, H4, H5, H7, H8, H31, H12, H14), in Markdown with LaTeX; final version, checked independently by AC4 (proof-check-report.md) and by the supervisor. Notation revised on 2026-11-17, content unchanged: y is the actual as a value and as a random quantity (was Y for the random actual), m the median of y in H5 to H7 (H6 and H7 wrote its logarithm m), s_+ the lowest expected s on the positive actuals in H8 (was m_+), h the density of epsilon in H11 (was g, the density of ln y elsewhere), two forecasts f_1 and f_2 in H10 (was f and g), the absolute distance d in H2 (was e), the shift tau in H6 (was t), lambda in H7 (was k), eta in H9 (was delta). proofs.md
  • A2 · proof The same proofs rendered as PDF, final version, with the notation revised on 2026-11-17 (see the proofs in Markdown). proofs.pdf
  • A3 · figure · diagram Figure 1 of the proofs: the weight y/(y+f)^2 of an actual on the forecast that minimises expected sMAPE, against y/f on a log scale, relative to its largest value, beside MAE's constant weight; drawn by claimset-code/smape-target/plot_weights.py. smape_weights.png
  • A4 · other The independent checker's reports on the proofs (Claude Fable 5.2, lab actor AC4): derivations from the statements alone, then comparisons with the proofs, with its counterexamples and the corrections they led to. proof-check-report.md
  • A5 · proof Proof of H35 (msMAPE beside sMAPE), in Markdown with LaTeX; checked independently by AC4 (CK26), whose suggested steps 4 and 7 were added after its check. Notation revised on 2026-11-17, content unchanged: the msMAPE term is written s~ (was m, now the median), its constant delta (was epsilon, now the random error of the other proofs), and |y|+|f| is l (was t). proof-msmape.md
  • A6 · data Every result of the lab record (15996 results of 10 runs) at full precision: the lab result's ID, its run, the quantity as the run's script named it, its conditions and its value (both as JSON). The evidence's tables give the values to four significant digits. smape-target.results.csv
  • A7 · lab_record Lab record of the study: questions, hypotheses, plans, runs, results and decisions, as an append-only, hash-chained event log (Claimset lab record specification v0.22, begun 2026-11-16; exported with claimset 0.44.3). lab.jsonl
  • A8 · data Every result of lab run RN1 (PL1: the targets of RMSE, MAE, MAPE and sMAPE on 34 distributions) at full precision, one row per result, sorted by setting: the lab result's ID, its run, the quantity, its conditions and its value (both as JSON). The evidence's tables give selected values to four significant digits. smape-target.RN1.results.csv
  • A9 · figure · data_plot inverse Gaussian shape=1000: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=1000.png). RN1-inverse_Gaussian_shape_1000.png
  • A10 · figure · data_plot inverse Gaussian shape=200: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=200.png). RN1-inverse_Gaussian_shape_200.png
  • A11 · figure · data_plot inverse Gaussian shape=100: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=100.png). RN1-inverse_Gaussian_shape_100.png
  • A12 · figure · data_plot inverse Gaussian shape=50: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=50.png). RN1-inverse_Gaussian_shape_50.png
  • A13 · figure · data_plot inverse Gaussian shape=20: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=20.png). RN1-inverse_Gaussian_shape_20.png
  • A14 · figure · data_plot inverse Gaussian shape=10: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/inverse_Gaussian_shape=10.png). RN1-inverse_Gaussian_shape_10.png
  • A15 · figure · data_plot log-skewed left sd=0.5 skew=-2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=0.5_skew=-2.png). RN1-log-skewed_left_sd_0.5_skew_-2.png
  • A16 · figure · data_plot log-skewed left sd=0.5 skew=-1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=0.5_skew=-1.png). RN1-log-skewed_left_sd_0.5_skew_-1.png
  • A17 · figure · data_plot log-skewed left sd=0.5 skew=-0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=0.5_skew=-0.5.png). RN1-log-skewed_left_sd_0.5_skew_-0.5.png
  • A18 · figure · data_plot log-skewed right sd=0.5 skew=+2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=0.5_skew=_2.png). RN1-log-skewed_right_sd_0.5_skew__2.png
  • A19 · figure · data_plot log-skewed right sd=0.5 skew=+1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=0.5_skew=_1.png). RN1-log-skewed_right_sd_0.5_skew__1.png
  • A20 · figure · data_plot log-skewed right sd=0.5 skew=+0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=0.5_skew=_0.5.png). RN1-log-skewed_right_sd_0.5_skew__0.5.p…
  • A21 · figure · data_plot lognormal sigma=0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=0.5.png). RN1-lognormal_sigma_0.5.png
  • A22 · figure · data_plot log-skewed left sd=1 skew=-2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=1_skew=-2.png). RN1-log-skewed_left_sd_1_skew_-2.png
  • A23 · figure · data_plot log-skewed left sd=1 skew=-1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=1_skew=-1.png). RN1-log-skewed_left_sd_1_skew_-1.png
  • A24 · figure · data_plot log-skewed left sd=1 skew=-0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_left_sd=1_skew=-0.5.png). RN1-log-skewed_left_sd_1_skew_-0.5.png
  • A25 · figure · data_plot log-skewed right sd=1 skew=+2: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=1_skew=_2.png). RN1-log-skewed_right_sd_1_skew__2.png
  • A26 · figure · data_plot log-skewed right sd=1 skew=+1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=1_skew=_1.png). RN1-log-skewed_right_sd_1_skew__1.png
  • A27 · figure · data_plot log-skewed right sd=1 skew=+0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/log-skewed_right_sd=1_skew=_0.5.png). RN1-log-skewed_right_sd_1_skew__0.5.png
  • A28 · figure · data_plot lognormal sigma=1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=1.png). RN1-lognormal_sigma_1.png
  • A29 · figure · data_plot bimodal equal modes: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/bimodal_equal_modes.png). RN1-bimodal_equal_modes.png
  • A30 · figure · data_plot bimodal left mode higher: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/bimodal_left_mode_higher.png). RN1-bimodal_left_mode_higher.png
  • A31 · figure · data_plot bimodal right mode higher: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/bimodal_right_mode_higher.png). RN1-bimodal_right_mode_higher.png
  • A32 · figure · data_plot normal straddling zero mean=0: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/normal_straddling_zero_mean=0.png). RN1-normal_straddling_zero_mean_0.png
  • A33 · figure · data_plot normal straddling zero mean=0.5: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/normal_straddling_zero_mean=0.5.png). RN1-normal_straddling_zero_mean_0.5.png
  • A34 · figure · data_plot normal straddling zero mean=1: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/normal_straddling_zero_mean=1.png). RN1-normal_straddling_zero_mean_1.png
  • A35 · figure · data_plot lognormal sigma=0.5 with zeros p0=0.25: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=0.5_with_zeros_p0=0.25.png). RN1-lognormal_sigma_0.5_with_zeros_p0_0…
  • A36 · figure · data_plot lognormal sigma=0.5 with zeros p0=0.42: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=0.5_with_zeros_p0=0.42.png). RN1-lognormal_sigma_0.5_with_zeros_p0_0…
  • A37 · figure · data_plot lognormal sigma=0.5 with zeros p0=0.47: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=0.5_with_zeros_p0=0.47.png). RN1-lognormal_sigma_0.5_with_zeros_p0_0…
  • A38 · figure · data_plot lognormal sigma=0.5 with zeros p0=0.6: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=0.5_with_zeros_p0=0.6.png). RN1-lognormal_sigma_0.5_with_zeros_p0_0…
  • A39 · figure · data_plot lognormal sigma=1 with zeros p0=0.25: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=1_with_zeros_p0=0.25.png). RN1-lognormal_sigma_1_with_zeros_p0_0.2…
  • A40 · figure · data_plot lognormal sigma=1 with zeros p0=0.37: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=1_with_zeros_p0=0.37.png). RN1-lognormal_sigma_1_with_zeros_p0_0.3…
  • A41 · figure · data_plot lognormal sigma=1 with zeros p0=0.42: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=1_with_zeros_p0=0.42.png). RN1-lognormal_sigma_1_with_zeros_p0_0.4…
  • A42 · figure · data_plot lognormal sigma=1 with zeros p0=0.6: for each measure (RMSE, MAE, MAPE, sMAPE), one violin per test-set size showing how the candidate the measure ranks first is spread over 1000 test sets, as its quantile level, with a boxplot of its median, quartiles and 5th and 95th percentiles and a diamond at its mean; a dashed line marks the measure's preferred candidate and dotted lines the other measures'. Drawn by lab run RN1 from its own results (lab/runs/RN1/figures/lognormal_sigma=1_with_zeros_p0=0.6.png). RN1-lognormal_sigma_1_with_zeros_p0_0.6…
  • A43 · data Every result of lab run RN2 (PL3: flatness of expected sMAPE as the spread of ln y grows) at full precision, one row per result, sorted by setting: the lab result's ID, its run, the quantity, its conditions and its value (both as JSON). The evidence's tables give selected values to four significant digits. smape-target.RN2.results.csv
  • A44 · figure · data_plot The quantile level of the sMAPE target against the spread sigma of ln y, for lognormal and log-Gumbel outcomes, beside the level of the log-mode: the target moves from the median to the mode of ln y as the spread grows. Drawn by lab run RN2 from its own results (lab/runs/RN2/figures/flatness_target.png). RN2-flatness_target.png
  • A45 · figure · data_plot Expected sMAPE against the quantile level of the forecast, one curve per spread sigma, for each family: the curve flattens towards 200 as sigma grows. Drawn by lab run RN2 from its own results (lab/runs/RN2/figures/flatness_expected_sMAPE.png). RN2-flatness_expected_sMAPE.png
  • A46 · figure · data_plot Lognormal outcomes: how much worse a forecast half a sigma or one sigma away from each measure's target scores, as a share of the score at the target, against sigma. Drawn by lab run RN2 from its own results (lab/runs/RN2/figures/flatness_sensitivity.png). RN2-flatness_sensitivity.png
  • A47 · figure · data_plot Ranking agreement: the share of test sets that prefer each measure's target to a forecast away from it, against sigma, for several test-set sizes. Drawn by lab run RN2 from its own results (lab/runs/RN2/figures/flatness_agreement.png). RN2-flatness_agreement.png
  • A48 · figure · data_plot Where the sMAPE minimiser of a test set falls, and how widely it spreads: its quantile level over test sets against sigma, for each family. Drawn by lab run RN2 from its own results (lab/runs/RN2/figures/flatness_minimiser.png). RN2-flatness_minimiser.png
  • A49 · data Every result of lab run RN7 (PL2: efficiency of the sMAPE minimiser against the sample median) at full precision, one row per result, sorted by setting: the lab result's ID, its run, the quantity, its conditions and its value (both as JSON). The evidence's tables give selected values to four significant digits. smape-target.RN7.results.csv
  • A50 · figure · data_plot lognormal outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its 99% band and the large-sample value as a dashed line; below 1 the sMAPE minimiser varies more than the sample median. Drawn by lab run RN7 from its own results (lab/runs/RN7/figures/efficiency_lognormal.png). RN7-efficiency_lognormal.png
  • A51 · figure · data_plot log-Laplace outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its 99% band and the large-sample value as a dashed line; below 1 the sMAPE minimiser varies more than the sample median. Drawn by lab run RN7 from its own results (lab/runs/RN7/figures/efficiency_log-Laplace.png). RN7-efficiency_log-Laplace.png
  • A52 · data Every result of lab run RN10 (PL11: the msMAPE and sMAPE targets) at full precision, one row per result, sorted by setting: the lab result's ID, its run, the quantity, its conditions and its value (both as JSON). The evidence's tables give selected values to four significant digits. smape-target.RN10.results.csv
  • A53 · figure · data_plot msMAPE beside sMAPE: near zero, the two targets as a share of the median against the share of zeros, for lognormal outcomes at medians 1 and 100, with the zero thresholds at median 1; away from zero, the relative difference of the msMAPE target from the sMAPE target for each distribution; on Poisson counts, the two targets against the mean. Drawn by lab run RN11 from the recorded results of RN10 (lab/runs/RN11/figures/msmape_targets.png). RN11-msmape_targets.png
  • A54 · code The study's code: the distributions, the exact quadrature of expected sMAPE, the formula checks, and the three simulations (targets.py, efficiency.py, flatness.py). smape-target

Score

Support of the paper: 98, Well supported

Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. The rubric. Every issue as text, for an agent preparing the next version.

The mean of its 10 principal claims; the lowest is C15 at 89. Open problems in the principal claims and what they rest on, each counted once, with the most it costs a claim:

  1. −6 Open problems Limitation L2 (could bias magnitude, not mitigated): Lab run RN7 deviated from plan PL2: RN7 ran efficiency.py at commit fa0ce62, not the plan's 92cd9a5: the script runs its 16 settings in parallel processes (one per core) instead of one after another; theory.py also gained a function (balance_exact) that efficiency.py does not use. Each (setting, test-set size) draws from its own seed as before, and on a subset the parallel code and the plan's code gave byte-identical results.json. Reason: Three earlier attempts on one core (over an hour) were killed by container restarts before the lab recorded them (notes NT2.R2, NT5, NT6); in parallel the run finished in about 20 minutes. The analysis, the settings, the seeds and the results are unchanged. In C15. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −5 Context A principal claim related to no earlier work. In C6. To remove it: Relate the claim to the earlier work it extends, addresses or departs from.
  3. −3 Plan The lab record flags the test: code changed after plan. In C15. To remove it: Rerun the test as planned, or explain the flag in a limitation.
  4. −1 Open problems Limitation L3 (reduces transferability, not mitigated): The test sets are independent draws from a fixed distribution. Real forecast errors are often correlated and their distribution can change over a test period, so a real test set carries less information than the same number of outcomes here. In C14, C15. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  5. −1 Open problems Limitation L4 (reduces transferability, not mitigated): Only the distribution families studied were computed; the formal claims (C4, C10) say what holds beyond them, the simulations only illustrate it. In C13, C14, C15, C16. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

Claims

C1: 100, Well supported

Novelty: extends earlier work, by the authors

Nothing lost.

C2: 100, Well supported

Novelty: new, by the authors

Nothing lost.

C3: 100, Well supported

Novelty: extends earlier work, by the authors

Nothing lost.

C4: 100, Well supported

Novelty: new, by the authors

Nothing lost.

C5: 100, Well supported

Novelty: replicates earlier work, by the authors

Nothing lost.

C6: 95, Well supported

Novelty: new, by the authors

  1. −5 Context A principal claim related to no earlier work. To remove it: Relate the claim to the earlier work it extends, addresses or departs from.

C7: 100, Well supported

Novelty: extends earlier work, by the authors

Nothing lost.

C8: 100, Well supported

Novelty: new, by the authors

Nothing lost.

C9: 100, Well supported

Novelty: extends earlier work, by the authors

Nothing lost.

C10: 100, Well supported

Novelty: new, by the authors

Nothing lost.

C11: 100, Well supported

Novelty: new, by the authors

Nothing lost.

C12: 100, Well supported

Novelty: new, by the authors

Nothing lost.

C13: 93, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L1 (could bias magnitude, not mitigated): Lab run RN1 deviated from plan PL1: RN1 ran at commit 3cb6c7c, not the plan's 92cd9a5. Between them only flatness.py and plot_flatness.py (plan PL3), test_smape.py and measures.py changed; in measures.py the unused function log_mae was removed (the supervisor dropped log-MAE from PL3, DC3). targets.py, theory.py, dists.py and the measures targets.py uses are unchanged. Reason: The commit of the PL3 revision came between PL1's plan and its run; it does not touch what PL1 runs. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −1 Open problems Limitation L4 (reduces transferability, not mitigated): Only the distribution families studied were computed; the formal claims (C4, C10) say what holds beyond them, the simulations only illustrate it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C14: 93, Well supported

Novelty: new, by the authors

  1. −5 Context Numbers in the statement match no recorded result. To remove it: Record each number as a result or table cell of the evidence the claim cites, and list it in the support's results; or correct the number.
  2. −1 Open problems Limitation L3 (reduces transferability, not mitigated): The test sets are independent draws from a fixed distribution. Real forecast errors are often correlated and their distribution can change over a test period, so a real test set carries less information than the same number of outcomes here. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  3. −1 Open problems Limitation L4 (reduces transferability, not mitigated): Only the distribution families studied were computed; the formal claims (C4, C10) say what holds beyond them, the simulations only illustrate it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C15: 89, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L2 (could bias magnitude, not mitigated): Lab run RN7 deviated from plan PL2: RN7 ran efficiency.py at commit fa0ce62, not the plan's 92cd9a5: the script runs its 16 settings in parallel processes (one per core) instead of one after another; theory.py also gained a function (balance_exact) that efficiency.py does not use. Each (setting, test-set size) draws from its own seed as before, and on a subset the parallel code and the plan's code gave byte-identical results.json. Reason: Three earlier attempts on one core (over an hour) were killed by container restarts before the lab recorded them (notes NT2.R2, NT5, NT6); in parallel the run finished in about 20 minutes. The analysis, the settings, the seeds and the results are unchanged. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −3 Plan The lab record flags the test: code changed after plan. To remove it: Rerun the test as planned, or explain the flag in a limitation.
  3. −1 Open problems Limitation L3 (reduces transferability, not mitigated): The test sets are independent draws from a fixed distribution. Real forecast errors are often correlated and their distribution can change over a test period, so a real test set carries less information than the same number of outcomes here. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  4. −1 Open problems Limitation L4 (reduces transferability, not mitigated): Only the distribution families studied were computed; the formal claims (C4, C10) say what holds beyond them, the simulations only illustrate it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C16: 99, Well supported

Novelty: new, by the authors

  1. −1 Open problems Limitation L4 (reduces transferability, not mitigated): Only the distribution families studied were computed; the formal claims (C4, C10) say what holds beyond them, the simulations only illustrate it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.

C17: 100, Well supported

Novelty: extends earlier work, by the authors

Nothing lost.

Not claimed

Planned proofs that give no claim of their own; the lab record has each with its reason.

  • Reworded, then proved again: H1 → C1, H2 → C2, H3 → C3, H4 → C4, H5 → C5, H18 → C6, H8 → C7, H12 → C8, H13 → C9, H19 → C10
  • Merged: H16 (inconclusive) → C4, H9 (inconclusive) → C3, H10 (inconclusive) → C2
  • Dropped: H11 (inconclusive)

Versions

  • Version 3 · 18 November 2026 at 13:08 UTC · updated
  • Version 2 (this page) · 17 November 2026 at 22:21 UTC · updated
  • Version 1 · 17 November 2026 at 15:01 UTC · active

Every version is kept in the public registry (github.com/entrylib/registry).