CLAIM C7 of entrylib:2026.000005 Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study
licence CC-BY-4.0; contributions C7

DEFINITIONS
D2 [method] claimset-submit: The claimset-submit skill of claimset 0.41.6 (claimset specification v0.41), together with `claimset lab export`, `claimset validate` and `claimset render`: it turns a lab record into a claimset and completes it for submission.
D3 [method] Sonnet agent: A Claude Code 2.1.322 subagent started with the Agent tool (type general-purpose) on the model claude-sonnet-5-5, working in a folder of its own that holds the study brief, `AGENTS.md` and the claimset skills as files; it asks the human by ending its turn with its questions. [part of D5]
D4 [method] Haiku agent: The same as the [[D3:Sonnet agent]], on the model claude-haiku-5-5. [part of D5]
D5 [method] agents under test: The two agent configurations compared: the [[D3:Sonnet agent]] and the [[D4:Haiku agent]]. [members D3, D4]
D6 [setting] ETS estimation brief: The research task given to every agent (`study/brief.md`): compare how well smooth 1.2.0 (`ES`) and statsforecast 2.1.1 (`AutoETS` with a fixed model form) estimate the smoothing parameters of the ETS model that generated simulated data, by bias, RMSE and MAE and by whether the RMSE falls as the series length grows; five processes (ETS(A,N,N) with α 0.1 and 0.5, ETS(A,A,N) with (α, β) (0.3, 0.05) and (0.5, 0.2), ETS(A,N,A) with m = 12 and (α, γ) (0.2, 0.1)), series lengths 50, 100, 250 and 500, 300 replicates per cell, smooth's default backcasting as a separate arm, failures counted.
D7 [design] lab phase: Phase 1: four runs, two of the [[D3:Sonnet agent]] and two of the [[D4:Haiku agent]], each in its own git repository with a local remote, given the same task ("Do the study in brief.md with claimset-lab, following its skill, from the first question to the finished record. Ask the human anything you need through me."). The orchestrating agent relays questions and answers them only from [[D9:the fixed answers]], gives no other help, and never shows an agent another run or [[D10:the answer key]]. Runs in parallel on one 4-core machine.
D8 [design] claimset phase: Phase 2: the lab record of run lab-sonnet-1 (the only one that verifies, planned before its data and fixed statsforecast's model form) copied to four folders, with the draft its agent had exported removed; two [[D3:Sonnet agent]] and two [[D4:Haiku agent]] runs, each told "Export this lab record with `claimset lab export` and complete the claimset for submission, following the claimset-submit skill. Do not submit it." Questions answered as in [[D7:the lab phase]].
D9 [design] fixed answers: The answers the orchestrating agent gives in place of the human, the same to every run (`study/human.md`): name and ORCID iD; licence CC-BY-4.0; funding none; the competing interest of the brief; "Approved." for a plan that names its quantities and decision rule before the data are run, otherwise "Please state the quantities and the decision rule first."; for highlights "Choose up to three principal claims you consider most important."; for anything else "Use your judgement and say what you decided." [part of D7, D8]
D10 [data] answer key: The findings a correct paper on [[D6:the brief]] would state, with their numbers (`reference/answer_key.md`), from a reference simulation of the full design written by the orchestrating agent before Phase 1 without claimset-lab (`reference/simulate.py`, `optcheck.py`, `damped_check.py`). Amended once during Phase 1 (statsforecast's β cap).
D16 [variable] output tokens: Model output tokens of an agent over its whole run, summed from the usage of every model call in its transcript. [unit 1]
D20 [variable] claimset validates: Whether `claimset validate` reports the final claimset valid (no errors).
D21 [variable] validator outcomes until valid: The position, among the validator outcomes a claimset run saw (the export's own summary included), of the first with no errors. [unit 1]
D22 [variable] warnings left: Validator warnings on the final claimset. [unit 1]
D23 [method] EntryLib rubric 0.2: The scoring rubric EntryLib applies to published claimsets (entrylib-rubric 0.2.0), run on this machine with `claimset score` and without `--novelty`: each principal claim starts at 100 and loses fixed costs for its problems (support, plan, checkability, context, open limitations and assumptions); the paper's score is that of its lowest principal claim.
D24 [variable] reliability score: The paper score of a claimset under [[D23:EntryLib rubric 0.2]], 0–100. [unit 1]
D25 [variable] claim correctness: Each claim of a claimset checked against [[D10:the answer key]]: `correct` (direction and magnitude agree within Monte Carlo error), `partly correct` (direction agrees; magnitude, scope or a qualifier does not), `wrong`, or `not supported by the record`.
D26 [variable] answer-key findings stated: Number of the answer key's findings on bias (K2–K5), statsforecast's non-convergence (K1.2) and statsforecast's β cap (K8) that a claimset states as claims. [unit 1]
D27 [variable] principal claims with scope: Number of a claimset's principal claims that have a scope. [unit 1]
D28 [variable] limitations: Number of limitations a claimset records. [unit 1]
D29 [variable] research agent's model kept: Whether a completed claimset keeps the model of the agent that did the research, as the lab record names it.
D30 [variable] interventions: Number of messages the orchestrating agent sent a run that were not answers from [[D9:the fixed answers]]. [unit 1]
D31 [design] reference simulation: The reference analysis behind [[D10:the answer key]]: 300 series per cell of [[D6:the brief]], each fitted by smooth with optimised initial states, smooth with backcasting and statsforecast `AutoETS(model, season_length, damped=False)` with its defaults (likelihood, bounds `both`), seeds `numpy.random.default_rng([20261111, process, T, replicate])`.
D32 [concept] statsforecast β cap: In statsforecast 2.1.1 (`statsforecast.ets.initparam`), when α is estimated its starting value is $$\alpha_0 = 0.0001 + 0.2 \cdot \frac{0.9999 - 0.0001}{m}$$ (0.20006 for $m = 1$), and the upper bound of β is set to $\min(0.9999, \alpha_0)$ for the whole optimisation, so β cannot exceed $\alpha_0$.
D37 [variable] statsforecast iteration cap: Number of statsforecast fits whose optimiser stopped at its iteration cap (`nit` = 1001) in [[D31:the reference simulation]]. [unit 1]

CLAIMS
C7 [descriptive; principal; support: empirical_quant E2 (E2.T1) artefacts A6, A8; assumes C21; uses C12; limitations L3, L4, L5; strength low (author); verifiability open; planning exploratory]
  None of the four claimsets that the [[D5:agents under test]] completed with [[D2:claimset-submit]] states as a claim any of the main findings of [[D10:the answer key]]: how large the estimation bias is, that statsforecast's optimiser fails to converge for the seasonal model, or that statsforecast caps β ([[D26:answer-key findings stated]]).
  scope: data_type: agent runs on one simulation study task | conditions: one lab record (lab-sonnet-1) completed four times; claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; claims only, not limitations or open questions | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
  addresses B6: Agent-written papers judged by an automated reviewer; here the check is which findings of a reference analysis the claims state.
C12 [descriptive; supporting; not_applicable; support: empirical_quant E2 artefacts A1; strength high (author); verifiability open; planning exploratory]
  The lab record that [[D8:the claimset phase]] started from holds three hypotheses, each a threshold test on the RMSE (a band for the ratio of the two packages, backcasting better in most cells at the shortest length, the longest length against the shortest), and no hypothesis about the size of the bias or the RMSE.
  scope: conditions: the lab record of run lab-sonnet-1, 1,130 events
C21 [descriptive; supporting; assumption (methodological, unchecked); verifiability not_applicable]
  [[D10:The answer key]] states the findings that a correct paper on [[D6:the brief]] would state, with correct numbers.

EVIDENCE
E2 [quant; design experiment; data_origin primary; n 4] [[D8:The claimset phase]]: two runs each of the [[D3:Sonnet agent]] and the [[D4:Haiku agent]] exported one lab record and completed the claimset with [[D2:claimset-submit]]. For each final claimset the orchestrating agent ran `claimset validate` and `claimset score` ([[D23:EntryLib rubric 0.2]]), counted the validator outcomes in the transcript (`tools/validate_runs.py`), checked that no record-derived field (hypothesis, planning, planned_at, test, exploration counts) had changed, and scored every claim against [[D10:the answer key]] (table T1). Table T2 scores, under the same rubric, the two claimsets that lab-phase agents wrote on their own records after finishing them (beyond their task).
  setup: studied D3, D4; cases D6; procedure D8, D9, D2; lens D10, D23, D20, D21, D22, D24, D25, D26, D27, D28, D29, D30, D16
  E2.T1 Outcomes of the four claimset runs of the claimset phase, all from the lab record of lab-sonnet-1
  | row | run | agent | D20 | D21 | D22 | D24 | D25 {outcome: correct} | D25 {outcome: partly correct} | D25 {outcome: wrong} | D26 | D27 | D28 | D29 | D30 | D16 |
  | r1 | claimset-sonnet-1 | D3 | yes | 4 | 2 | 73 | 4 | 1 | 0 | 0 | 3 | 3 | yes | 0 | 50493 |
  | r2 | claimset-sonnet-2 | D3 | yes | 5 | 3 | 72 | 5 | 0 | 0 | 0 | 3 | 3 | yes | 0 | 44792 |
  | r3 | claimset-haiku-1 | D4 | yes | 6 | 1 | 79 | 4 | 1 | 0 | 0 | 0 | 3 | no | 0 | 69641 |
  | r4 | claimset-haiku-2 | D4 | yes | 4 | 11 | 78 | 3 | 2 | 0 | 0 | 0 | 0 | yes | 1 | 55371 |
  E2.T2 EntryLib rubric 0.2 scores of the two claimsets written by lab-phase agents on their own records
  | row | run | agent | planning of the principal claims | D24 | D28 | D27 |
  | r1 | lab-sonnet-2 | D3 | confirmatory | 73 | 4 | 3 |
  | r2 | lab-haiku-1 | D4 | exploratory | 92 | 0 | 3 |
E3 [simulation; data_origin primary; runs 300] [[D31:The reference simulation]] of [[D6:the brief]] (R = 300 per cell, 20 cells, three arms, 110 s on 4 cores), with `optcheck.py` and `damped_check.py`. Table T1: statsforecast's β estimates for ETS(A,A,N) with (α, β) = (0.5, 0.2), against smooth's; table T2: statsforecast fits of ETS(A,N,A) that stopped at the optimiser's iteration cap. The β cap was first reported by run lab-haiku-1 (its exploratory run RN1); it was verified here against the statsforecast 2.1.1 source and these estimates, and added to the answer key.
  setup: studied D31; cases D6; procedure D31; lens D32, D37
  E3.T1 statsforecast and smooth (optimised initial states) estimates of β for ETS(A,A,N) with α = 0.5, β = 0.2, 300 series per length
  | row | series length | largest statsforecast β estimate | statsforecast fits within 0.001 of 0.20006 | smooth fits with β above 0.20006 |
  | r1 | 50 | 0.20006 | 100 | 131 |
  | r2 | 100 | 0.20006 | 135 | 146 |
  | r3 | 250 | 0.20006 | 132 | 132 |
  | r4 | 500 | 0.20006 | 144 | 137 |
  E3.T2 statsforecast fits of ETS(A,N,A) (α = 0.2, γ = 0.1, m = 12) at the optimiser's iteration cap, of 300 per length
  | row | series length | D37 |
  | r1 | 50 | 288 |
  | r2 | 100 | 296 |
  | r3 | 250 | 300 |
  | r4 | 500 | 299 |
  parameters: A6
  seeds: ["20261111"]

LIMITATIONS
L3 [measurement; affects C6, C7; could_bias_magnitude; mitigation partial] One scorer, the orchestrating agent, wrote the answer key and the protocol and scored every claim, knowing which agent wrote it.
L4 [deviation; affects C6, C7, C20; could_bias_magnitude; mitigation none] The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key.
L5 [sampling; affects C1, C2, C3, C4, C15, C5, C6, C7, C8, C9, C10, C11; reduces_transferability; mitigation none] One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322).

ARTEFACTS
A1 [data] The eight runs as the agents left them: each folder's files without its git directory, and its full git history (study branch included) in repo.bundle.
A2 [trace] The eight agents' transcripts (Claude Code subagent JSONL), with every tool call and result and the usage of every model call.
A3 [prompt] The task prompts given to the agents in each phase.
A6 [data] The reference analysis: estimates.csv, summary.csv, consistency.csv, optcheck.csv, damped_check.csv and answer_key.md.
A7 [code] The reference analysis scripts simulate.py, optcheck.py and damped_check.py.
A8 [data] Scoring per run, rubric scores as JSON, summary.md and report.md.
A9 [code] The scoring scripts transcript.py, lab_summary.py and validate_runs.py.
A10 [software] EntryLib's scoring rubric, entrylib-rubric 0.2.0 (src/rubric/rubric.json holds the costs; src/rubric/score.py applies them).

HOW TO CITE
This claim: Svetunkov, I. (guarantor) (2026). Claim C7, in: Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study. EntryLib. entrylib:2026.000005@v2#C7. https://entrylib.org/p/2026.000005/C7
The claimset it is from: Svetunkov, I. (guarantor) (2026). Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study. EntryLib. entrylib:2026.000005@v2. https://entrylib.org/p/2026.000005@v2