LAB RECORD: Which forecast does each error measure reward? A simulation study of the functionals targeted by RMSE, MAE, MAPE and sMAPE Study: measure-rankings · Lab record specification v0.22 · Begun 2026-11-15 · Repository: https://github.com/entrylib/study-planning Events: 49; runs: 2; results: 10,546 (kept in files, never listed here); head: seq 10595, sha256:0bdd57230f6454bba0a51a572a0ee71e1c65226834bfe4e96ee4c7909613808e This view is made from the log; the log is the record (lab record §11). WHO - AC1 Claude Code (agent; Anthropic Claude Opus 5.5; in Claude Code; simulations), main log; credit: conceptualization, methodology, formal_analysis - AC2 Ivan Svetunkov (human; ORCID 0000-0001-7826-0281), main log; credit: supervision QUESTIONS AND HYPOTHESES Q1 When candidate point forecasts are fixed before any test data are seen, which candidate does each error measure (RMSE, MAE, MAPE, sMAPE) rank first, and how of… TIMELINE (every event but results, in log order: seq, time, ID, type, writer) 1 2026-11-15 10:16 INIT init by -: began "Which forecasts do error measures rank first, and how long must the test set be?" under lab record v0.22 2 2026-11-15 10:17 AC1 actor by AC1: registers Claude Code (agent) 3 2026-11-15 11:04 AC2 actor by AC1: registers Ivan Svetunkov (human, ORCID 0000-0001-7826-0281) 4 2026-11-15 11:05 Q1 question by AC1: When candidate point forecasts are fixed before any test data are seen, which candidate does each error measure (RMSE, MAE, MAPE, sMAPE) rank first, and how of… 5 2026-11-15 11:05 TM1 term by AC1: RMSE [variable] 6 2026-11-15 11:05 TM2 term by AC1: MAE [variable] 7 2026-11-15 11:05 TM3 term by AC1: MAPE [variable] 8 2026-11-15 11:05 TM4 term by AC1: sMAPE [variable] 9 2026-11-15 11:05 TM5 term by AC1: candidate forecast [concept] 10 2026-11-15 11:05 TM6 term by AC1: expected score [concept] 11 2026-11-15 11:05 TM7 term by AC1: preferred candidate [concept] 12 2026-11-15 11:05 TM8 term by AC1: ranking agreement [concept] 13 2026-11-15 11:05 NT1 note by AC1: Scope, agreed with the supervisor (AC2) on 2026-11-15, before any design or run. What this study does: it fixes candidate point forecasts in advance from the t… 14 2026-11-15 14:46 DC1 decision by AC1: The supervisor (AC2), 2026-11-15: no statistical test (such as Friedman with Nemenyi) between candidate forecasts. In a simulation the number of test sets is c… 15 2026-11-15 14:46 DC2 decision by AC1: The supervisor (AC2), 2026-11-15: no forecasting model (ETS(A,N,N) is dropped): outcomes are generated directly from known distributions, and the candidates ar… 16 2026-11-15 14:46 DC3 decision by AC1: The supervisor (AC2), 2026-11-15: continuous outcomes are positive. Normal and Laplace outcomes are placed at a level high enough relative to their spread that… 17 2026-11-15 14:46 TM9 term by AC1: MAPE [variable] 18 2026-11-15 14:46 TM10 term by AC1: selection share [concept] 19 2026-11-15 14:46 NT2 note by AC1: Scope, agreed with the supervisor (AC2) on 2026-11-15, before any design or run (restated after DC1 to DC3). What this study does: it fixes candidate point for… 20 2026-11-15 14:46 DC4 decision by AC1: The supervisor (AC2), 2026-11-15: no statistical test (such as Friedman with Nemenyi) between candidate forecasts. In a simulation the number of test sets is c… 21 2026-11-15 14:55 TM11 term by AC1: symmetric positive outcomes [data] 22 2026-11-15 14:55 TM12 term by AC1: skewed positive outcomes [data] 23 2026-11-15 14:55 TM13 term by AC1: outcomes with outliers [data] 24 2026-11-15 14:55 TM14 term by AC1: count outcomes [data] 25 2026-11-15 14:55 TM15 term by AC1: intermittent outcomes [data] 26 2026-11-15 14:55 TM16 term by AC1: selection design [design] 27 2026-11-15 14:56 PL1 plan by AC1: [estimation, requires approval] for Q1: The selection design over 24 outcome distributions: symmetric positive (normal, Laplace), skewed positive (lognormal s… 28 2026-11-15 18:18 PL2 plan by AC1: [estimation, requires approval] for Q1: The selection design over 24 outcome distributions: symmetric positive (normal, Laplace), skewed positive (lognormal s… 29 2026-11-15 19:16 TM17 term by AC1: symmetric positive outcomes [data] 30 2026-11-15 19:16 DC5 decision by AC1: The supervisor (AC2), 2026-11-15: normal and Laplace outcomes at several scales ("Try more scale values for normal and laplace"): standard deviations 1, 5, 10,… 31 2026-11-15 19:16 PL3 plan by AC1: [estimation, requires approval] for Q1: The selection design over 28 outcome distributions: symmetric positive (normal with standard deviation 1, 5, 10, 15 and… 32 2026-11-15 20:14 PL4 plan by AC1: [estimation, requires approval] for Q1: The selection design over 28 outcome distributions: symmetric positive (normal with standard deviation 1, 5, 10, 15 and… 33 2026-11-15 20:15 AP1 approval by AC1: PL4 approved by AC2 at 2026-11-15T20:15:05Z, recorded by agent AC1: "We don't need 100000. The rest is fine. Go" 34 2026-11-15 20:34 RN1 run by AC1: [confirmatory, success, under PL4, commit a98e5f5] python3 claimset-code/measure-rankings/simulate.py --seed 1 --ref-seed 1001 --reps-small 1000; 10,384 results in runs/RN1/results.json (RS1 to RS10384) 10419 2026-11-16 08:04 RN2 run by AC1: [exploratory, success, commit 9759b03] python3 claimset-code/measure-rankings/explore_smape.py --seed 7; 162 results in runs/RN2/results.json (RS10385 to RS10546) 10582 2026-11-16 09:59 DC6 decision by AC1: The supervisor (AC2), 2026-11-16, after the exploratory run RN2: a second estimation plan, with the same design as PL1, adds inverse Gaussian outcomes (long ri… 10583 2026-11-16 09:59 TM18 term by AC1: MAPE [variable] 10584 2026-11-16 09:59 TM19 term by AC1: inverse Gaussian outcomes [data] 10585 2026-11-16 09:59 TM20 term by AC1: log-skewed outcomes [data] 10586 2026-11-16 09:59 TM21 term by AC1: bimodal outcomes [data] 10587 2026-11-16 10:00 TM22 term by AC1: outcomes straddling zero [data] 10588 2026-11-16 10:00 PL5 plan by AC1: [estimation, requires approval] for Q1: The selection design of PL1, on a second set of 26 outcome distributions: inverse Gaussian (mean 100, shape 1000, 200,… 10589 2026-11-16 10:41 DC7 decision by AC1: The supervisor (AC2), 2026-11-16: this study is finalised and submitted on the results of PL1.R4 (run RN1). PL5 (not approved, never run) moves to a separate s… 10590 2026-11-16 11:01 TM23 term by AC1: selection design [design] 10591 2026-11-16 11:01 TM24 term by AC1: expected score [concept] 10592 2026-11-16 11:01 TM25 term by AC1: candidate forecast [concept] 10593 2026-11-16 11:01 TM26 term by AC1: MAPE [variable] 10594 2026-11-16 11:10 TM27 term by AC1: selection design [design] 10595 2026-11-16 11:10 TM28 term by AC1: expected score [concept]