Claim C15 · descriptive · entrylib:2026.000013@v3 · 2026-11-18

Corrected · 18 November 2026 at 13:08 UTC

Version 3 answers a reader's comments. Corrected: C3 no longer says sMAPE is minimised by the median for small errors, only near it, with a bound; C4 keeps the weighted median, and C12 now holds what was said about the ordinary median; the measure called msMAPE in version 2 is the Suilin loss. Added: how well a test set tells two forecasts apart as the actuals spread out (C18); the large-sample distribution of the sMAPE minimiser (C19); Smyl's msMAPE (C20) and what it rewards on intermittent demand (C21). Also added: sections, the definition of sMAPE first, and a limitation on C6.

By Ivan Svetunkov (ORCID 0000-0001-7826-0281)

What does sMAPE reward? An analytical and simulation study · C15

Principal claim C15 of this paper (Svetunkov, I. (guarantor), 2026), with what it rests on. Read the whole paper.

Text file for AI

Text file: the claim with what it rests on as plain text, to attach when your assistant cannot open the address.

The message

Read https://entrylib.org/p/2026.000013/C15, one claim of a research paper with the evidence it rests on. Tell me briefly what it says and how well it is supported. If you cannot open the link, tell me instead of guessing. Then wait for my questions.

The claim

SupportS 89C15 · descriptive · planned estimate · novelty: new, by the authors

Where sMAPE and MAE both target the median (log-symmetric outcomes), the sMAPE minimiser of a test set varies more than the sample median for every spread and test-set size studied: its relative efficiency is below 1 and falls as the actuals spread out, at n=10000n = 10000 from 0.97 to 0.03 on lognormal actuals as σ\sigma grows from 0.25 to 6, and from 1.00 to 0.34 on log-Laplace actuals as bb grows from 0.25 to 6, close to the large-sample values.

Evidence

E15 · quant · other · cited by C15

Lab runs RN7 under plan PL2. Design: Efficiency design (D17) on 16 log-symmetric distributions (D26): lognormal sigma and log-Laplace b, each 0.25, 0.5, 1, 1.5, 2, 3, 4, 6. Test sets of n = 6, 8, 12, 13, 14, 18, 24, 36, 48, 100, 250, 500, 1000 and 10000 outcomes, 10000 test sets per size, seed 4 with the setting's index and n. In each test set: the sample median and the exact sMAPE minimiser (D8). From each distribution, derived (D15): the large-sample n Var of ln of each, from H11, and their ratio. python3 claimset-code/smape-target/efficiency.py --seed 4 --reps 10000 (numpy 2.5.3, matplotlib 3.11.2). Analysis: Per distribution and n, over test sets: the variances of ln(sample median) and ln(sMAPE minimiser); the relative efficiency (D9) with a 99% bootstrap percentile interval (2000 resamples of test sets); the ratio of their mean squared errors about the true median; the mean of ln(sMAPE minimiser) minus ln(median). Per distribution: the asymptotic relative efficiency. Figure: relative efficiency over n, a panel per distribution, with its band, the asymptotic value dashed and 1 dotted. Compared with H11 and H14 (iii); no test is made and no threshold is set.

lognormal outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its 9…
A50 lognormal outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its 99% band and the large-sample value as a dashed line; below 1 the sMAPE minimiser varies more than the sample median. Drawn by lab run RN7 from its own results (lab/runs/RN7/figures/efficiency_lognormal.png).
log-Laplace outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its…
A51 log-Laplace outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its 99% band and the large-sample value as a dashed line; below 1 the sMAPE minimiser varies more than the sample median. Drawn by lab run RN7 from its own results (lab/runs/RN7/figures/efficiency_log-Laplace.png).
E15.T1 · Relative efficiency of the sMAPE minimiser of a test set against the sample median, at test-set sizes 6, 12, 100, 1000 and 10000, beside its large-sample value
Rowsettingrelative efficiency (test-set size: 6)relative efficiency (test-set size: 12)relative efficiency (test-set size: 100)relative efficiency (test-set size: 1000)relative efficiency (test-set size: 10000)asymptotic relative efficiency
r1lognormal sigma=0.250.60590.70260.89960.96620.97010.9696
r2lognormal sigma=0.50.6110.70830.88020.8850.89060.8881
r3lognormal sigma=10.60670.65550.65240.66590.65650.6601
r4lognormal sigma=1.50.54840.51230.45950.46330.450.454
r5lognormal sigma=20.5010.43920.3230.30770.30430.3073
r6lognormal sigma=30.43810.36460.20580.15570.15130.1474
r7lognormal sigma=40.41480.32720.15610.0890.07890.0778
r8lognormal sigma=60.39330.30270.12290.05080.03170.0279
r9log-Laplace b=0.250.70490.82680.97690.99180.99690.9969
r10log-Laplace b=0.50.69890.8190.94480.97790.97530.9765
r11log-Laplace b=10.6670.68870.84790.87630.88530.8953
r12log-Laplace b=1.50.57660.59470.69890.77890.78040.8035
r13log-Laplace b=20.51790.50360.6120.6830.70790.7206
r14log-Laplace b=30.46720.43650.45860.52570.56280.5902
r15log-Laplace b=40.43530.37640.34960.42670.48030.4968
r16log-Laplace b=60.41670.34920.26670.28960.34410.3753
E15.T2 · Bias and variance at test-set sizes 12 and 1000, for spreads 0.25, 1 and 6: the mean of ln(sMAPE minimiser) minus ln(median), and the variances of ln(sample median) and ln(sMAPE minimiser); every setting and size is in the run's CSV
Rowsettingtest-set sizemean of ln(sMAPE minimiser) minus ln(median)variance of ln(sample median)variance of ln(sMAPE minimiser)ratio of mean squared errors about the true median
r1lognormal sigma=0.25120.00080.00720.01030.7026
r2lognormal sigma=0.2510000.00000.00010.00010.9662
r3lognormal sigma=1120.00080.11590.17680.6555
r4lognormal sigma=110000.00010.00160.00240.6659
r5lognormal sigma=6120.00214.28714.170.3027
r6lognormal sigma=61000-0.02090.05711.1260.0507
r7log-Laplace b=0.2512-0.00040.00730.00880.8268
r8log-Laplace b=0.2510000.00020.00010.00010.9918
r9log-Laplace b=1120.00140.11910.1730.6887
r10log-Laplace b=110000.00030.00110.00120.8764
r11log-Laplace b=6120.02124.26712.220.3492
r12log-Laplace b=610000.0030.03770.13030.2896

Limitations

L2 · analytic flexibility · could bias magnitude · affects C15 · mitigation: none

Lab run RN7 deviated from plan PL2: RN7 ran efficiency.py at commit fa0ce62, not the plan's 92cd9a5: the script runs its 16 settings in parallel processes (one per core) instead of one after another; theory.py also gained a function (balance_exact) that efficiency.py does not use. Each (setting, test-set size) draws from its own seed as before, and on a subset the parallel code and the plan's code gave byte-identical results.json. Reason: Three earlier attempts on one core (over an hour) were killed by container restarts before the lab recorded them (notes NT2.R2, NT5, NT6); in parallel the run finished in about 20 minutes. The analysis, the settings, the seeds and the results are unchanged.

Definitions

Methods 1

D17 efficiency design: For each outcome distribution and test-set size nn: draw test sets of nn independent outcomes; in each, compute the sample median and the exact sMAPE minimiser (an outcome of the test set, since the test set's sMAPE is concave in ln⁡f\ln f between consecutive outcomes); over test sets, compare the variances of their logarithms.

Variables and measures 3

D11 MAE: Mean absolute error of a point forecast ff over a test set of actuals y1,…,yny_1, \dots, y_n: 1n∑t|yt−f|\frac1n\sum_t |y_t-f|.

D8 sMAPE minimiser of a test set: The forecast that minimises the sMAPE of one test set y1,…,yny_1, \dots, y_n, f^n=arg⁡minf⁡1n∑ts(yt,f)\hat f_n = \arg\min_f \frac1n\sum_t s(y_t, f), found numerically (the sample counterpart of the sMAPE target).

D9 relative efficiency: For a test-set size nn and an outcome distribution where sMAPE and MAE both target the median: RE(n)=Var⁡(ln⁡m^n)/Var⁡(ln⁡f^n)\mathrm{RE}(n) = \operatorname{Var}(\ln \hat m_n) / \operatorname{Var}(\ln \hat f_n) over test sets, where m^n\hat m_n is the sample median (the MAE minimiser) and f^n\hat f_n the sMAPE minimiser of the same test set. Above 1, sMAPE's minimiser varies less; below 1, MAE's does.

Populations 1

D26 log-symmetric outcomes: Lognormal outcomes with σ\sigma, and log-Laplace outcomes with scale bb, each 0.25, 0.5, 1, 1.5, 2, 3, 4 or 6, median 100: ln⁡y\ln y symmetric and strictly unimodal, so that sMAPE and MAE both target the median (C5).

Concepts 2

D6 sMAPE target: The value or values of ff that minimise expected sMAPE for a given distribution of the actual yy, arg⁡minf⁡E[s(y,f)]\arg\min_f \mathrm E[s(y,f)]: the functional sMAPE rewards. It may be a single value or several.

D15 derived value: A value computed from the outcome distribution itself by numerical integration of its density (Gauss-Legendre quadrature and a convolution on a grid of 524288 points of ln⁡y\ln y), never from a sample: the expected scores, the local minima of expected sMAPE, the formulas of the hypotheses.

Context

Supports B7 · from C15
C11 found sMAPE's choices in short test sets spreading more than MAE's; C15 measures the loss of precision for test sets of 6 to 10000 outcomes.
Supports B6 · from C15
C6 recommends against sMAPE for point forecasts; C15 shows that even where it targets the median it estimates it less precisely than the sample median.
Supports B13 · from C15
Kolassa (2026) concludes that sMAPE should not be used; C15 shows that even where it targets the median it estimates it less precisely than the sample median.

Artefacts

  • A6 · data Every result of the lab record (17182 results of 17 runs) at full precision: the lab result's ID, its run, the quantity as the run's script named it, its conditions and its value (both as JSON). The evidence's tables give the values to four significant digits. smape-target.results.csv
  • A49 · data Every result of lab run RN7 (PL2: efficiency of the sMAPE minimiser against the sample median) at full precision, one row per result, sorted by setting: the lab result's ID, its run, the quantity, its conditions and its value (both as JSON). The evidence's tables give selected values to four significant digits. smape-target.RN7.results.csv
  • A50 · figure · data_plot lognormal outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its 99% band and the large-sample value as a dashed line; below 1 the sMAPE minimiser varies more than the sample median. Drawn by lab run RN7 from its own results (lab/runs/RN7/figures/efficiency_lognormal.png). RN7-efficiency_lognormal.png
  • A51 · figure · data_plot log-Laplace outcomes: relative efficiency of the sMAPE minimiser of a test set against the sample median, against test-set size, one panel per spread, with its 99% band and the large-sample value as a dashed line; below 1 the sMAPE minimiser varies more than the sample median. Drawn by lab run RN7 from its own results (lab/runs/RN7/figures/efficiency_log-Laplace.png). RN7-efficiency_log-Laplace.png

Score

Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.

C15: 89, Well supported

Novelty: new, by the authors

  1. −6 Open problems Limitation L2 (could bias magnitude, not mitigated): Lab run RN7 deviated from plan PL2: RN7 ran efficiency.py at commit fa0ce62, not the plan's 92cd9a5: the script runs its 16 settings in parallel processes (one per core) instead of one after another; theory.py also gained a function (balance_exact) that efficiency.py does not use. Each (setting, test-set size) draws from its own seed as before, and on a subset the parallel code and the plan's code gave byte-identical results.json. Reason: Three earlier attempts on one core (over an hour) were killed by container restarts before the lab recorded them (notes NT2.R2, NT5, NT6); in parallel the run finished in about 20 minutes. The analysis, the settings, the seeds and the results are unchanged. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  2. −3 Plan The lab record flags the test: code changed after plan. To remove it: Rerun the test as planned, or explain the flag in a limitation.
  3. −1 Open problems Limitation L3 (reduces transferability, not mitigated): The test sets are independent draws from a fixed distribution. Real forecast errors are often correlated and their distribution can change over a test period, so a real test set carries less information than the same number of outcomes here. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
  4. −1 Open problems Limitation L4 (reduces transferability, not mitigated): Only the distribution families studied were computed; the formal claims (C4, C10) say what holds beyond them, the simulations only illustrate it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.