CLAIM C9 of entrylib:2026.000005 Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study
licence CC-BY-4.0; contributions C9

DEFINITIONS
D2 [method] claimset-submit: The claimset-submit skill of claimset 0.41.6 (claimset specification v0.41), together with `claimset lab export`, `claimset validate` and `claimset render`: it turns a lab record into a claimset and completes it for submission.
D3 [method] Sonnet agent: A Claude Code 2.1.322 subagent started with the Agent tool (type general-purpose) on the model claude-sonnet-5-5, working in a folder of its own that holds the study brief, `AGENTS.md` and the claimset skills as files; it asks the human by ending its turn with its questions. [part of D5]
D4 [method] Haiku agent: The same as the [[D3:Sonnet agent]], on the model claude-haiku-5-5. [part of D5]
D5 [method] agents under test: The two agent configurations compared: the [[D3:Sonnet agent]] and the [[D4:Haiku agent]]. [members D3, D4]
D6 [setting] ETS estimation brief: The research task given to every agent (`study/brief.md`): compare how well smooth 1.2.0 (`ES`) and statsforecast 2.1.1 (`AutoETS` with a fixed model form) estimate the smoothing parameters of the ETS model that generated simulated data, by bias, RMSE and MAE and by whether the RMSE falls as the series length grows; five processes (ETS(A,N,N) with α 0.1 and 0.5, ETS(A,A,N) with (α, β) (0.3, 0.05) and (0.5, 0.2), ETS(A,N,A) with m = 12 and (α, γ) (0.2, 0.1)), series lengths 50, 100, 250 and 500, 300 replicates per cell, smooth's default backcasting as a separate arm, failures counted.
D7 [design] lab phase: Phase 1: four runs, two of the [[D3:Sonnet agent]] and two of the [[D4:Haiku agent]], each in its own git repository with a local remote, given the same task ("Do the study in brief.md with claimset-lab, following its skill, from the first question to the finished record. Ask the human anything you need through me."). The orchestrating agent relays questions and answers them only from [[D9:the fixed answers]], gives no other help, and never shows an agent another run or [[D10:the answer key]]. Runs in parallel on one 4-core machine.
D8 [design] claimset phase: Phase 2: the lab record of run lab-sonnet-1 (the only one that verifies, planned before its data and fixed statsforecast's model form) copied to four folders, with the draft its agent had exported removed; two [[D3:Sonnet agent]] and two [[D4:Haiku agent]] runs, each told "Export this lab record with `claimset lab export` and complete the claimset for submission, following the claimset-submit skill. Do not submit it." Questions answered as in [[D7:the lab phase]].
D9 [design] fixed answers: The answers the orchestrating agent gives in place of the human, the same to every run (`study/human.md`): name and ORCID iD; licence CC-BY-4.0; funding none; the competing interest of the brief; "Approved." for a plan that names its quantities and decision rule before the data are run, otherwise "Please state the quantities and the decision rule first."; for highlights "Choose up to three principal claims you consider most important."; for anything else "Use your judgement and say what you decided." [part of D7, D8]
D10 [data] answer key: The findings a correct paper on [[D6:the brief]] would state, with their numbers (`reference/answer_key.md`), from a reference simulation of the full design written by the orchestrating agent before Phase 1 without claimset-lab (`reference/simulate.py`, `optcheck.py`, `damped_check.py`). Amended once during Phase 1 (statsforecast's β cap).
D16 [variable] output tokens: Model output tokens of an agent over its whole run, summed from the usage of every model call in its transcript. [unit 1]
D20 [variable] claimset validates: Whether `claimset validate` reports the final claimset valid (no errors).
D21 [variable] validator outcomes until valid: The position, among the validator outcomes a claimset run saw (the export's own summary included), of the first with no errors. [unit 1]
D22 [variable] warnings left: Validator warnings on the final claimset. [unit 1]
D23 [method] EntryLib rubric 0.2: The scoring rubric EntryLib applies to published claimsets (entrylib-rubric 0.2.0), run on this machine with `claimset score` and without `--novelty`: each principal claim starts at 100 and loses fixed costs for its problems (support, plan, checkability, context, open limitations and assumptions); the paper's score is that of its lowest principal claim.
D24 [variable] reliability score: The paper score of a claimset under [[D23:EntryLib rubric 0.2]], 0–100. [unit 1]
D25 [variable] claim correctness: Each claim of a claimset checked against [[D10:the answer key]]: `correct` (direction and magnitude agree within Monte Carlo error), `partly correct` (direction agrees; magnitude, scope or a qualifier does not), `wrong`, or `not supported by the record`.
D26 [variable] answer-key findings stated: Number of the answer key's findings on bias (K2–K5), statsforecast's non-convergence (K1.2) and statsforecast's β cap (K8) that a claimset states as claims. [unit 1]
D27 [variable] principal claims with scope: Number of a claimset's principal claims that have a scope. [unit 1]
D28 [variable] limitations: Number of limitations a claimset records. [unit 1]
D29 [variable] research agent's model kept: Whether a completed claimset keeps the model of the agent that did the research, as the lab record names it.
D30 [variable] interventions: Number of messages the orchestrating agent sent a run that were not answers from [[D9:the fixed answers]]. [unit 1]

CLAIMS
C9 [descriptive; principal; support: empirical_quant E2 (E2.T1) artefacts A1; limitations L1, L2, L5; strength low (author); verifiability open; planning exploratory]
  The two claimsets that the [[D4:Haiku agent]] completed give a scope to none of their three principal claims, against all three in each of the two that the [[D3:Sonnet agent]] completed ([[D27:principal claims with scope]]).
  scope: data_type: agent runs on one simulation study task | conditions: one lab record (lab-sonnet-1) completed four times; claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; principal claims only | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields

EVIDENCE
E2 [quant; design experiment; data_origin primary; n 4] [[D8:The claimset phase]]: two runs each of the [[D3:Sonnet agent]] and the [[D4:Haiku agent]] exported one lab record and completed the claimset with [[D2:claimset-submit]]. For each final claimset the orchestrating agent ran `claimset validate` and `claimset score` ([[D23:EntryLib rubric 0.2]]), counted the validator outcomes in the transcript (`tools/validate_runs.py`), checked that no record-derived field (hypothesis, planning, planned_at, test, exploration counts) had changed, and scored every claim against [[D10:the answer key]] (table T1). Table T2 scores, under the same rubric, the two claimsets that lab-phase agents wrote on their own records after finishing them (beyond their task).
  setup: studied D3, D4; cases D6; procedure D8, D9, D2; lens D10, D23, D20, D21, D22, D24, D25, D26, D27, D28, D29, D30, D16
  E2.T1 Outcomes of the four claimset runs of the claimset phase, all from the lab record of lab-sonnet-1
  | row | run | agent | D20 | D21 | D22 | D24 | D25 {outcome: correct} | D25 {outcome: partly correct} | D25 {outcome: wrong} | D26 | D27 | D28 | D29 | D30 | D16 |
  | r1 | claimset-sonnet-1 | D3 | yes | 4 | 2 | 73 | 4 | 1 | 0 | 0 | 3 | 3 | yes | 0 | 50493 |
  | r2 | claimset-sonnet-2 | D3 | yes | 5 | 3 | 72 | 5 | 0 | 0 | 0 | 3 | 3 | yes | 0 | 44792 |
  | r3 | claimset-haiku-1 | D4 | yes | 6 | 1 | 79 | 4 | 1 | 0 | 0 | 0 | 3 | no | 0 | 69641 |
  | r4 | claimset-haiku-2 | D4 | yes | 4 | 11 | 78 | 3 | 2 | 0 | 0 | 0 | 0 | yes | 1 | 55371 |
  E2.T2 EntryLib rubric 0.2 scores of the two claimsets written by lab-phase agents on their own records
  | row | run | agent | planning of the principal claims | D24 | D28 | D27 |
  | r1 | lab-sonnet-2 | D3 | confirmatory | 73 | 4 | 3 |
  | r2 | lab-haiku-1 | D4 | exploratory | 92 | 0 | 3 |

LIMITATIONS
L1 [sample_size; affects C2, C3, C4, C8, C9, C17, C18; could_reverse; mitigation none] Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs.
L2 [confounding; affects C1, C2, C3, C5, C9; could_bias_magnitude; mitigation none] The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine.
L5 [sampling; affects C1, C2, C3, C4, C15, C5, C6, C7, C8, C9, C10, C11; reduces_transferability; mitigation none] One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322).

ARTEFACTS
A1 [data] The eight runs as the agents left them: each folder's files without its git directory, and its full git history (study branch included) in repo.bundle.
A2 [trace] The eight agents' transcripts (Claude Code subagent JSONL), with every tool call and result and the usage of every model call.
A3 [prompt] The task prompts given to the agents in each phase.
A6 [data] The reference analysis: estimates.csv, summary.csv, consistency.csv, optcheck.csv, damped_check.csv and answer_key.md.
A8 [data] Scoring per run, rubric scores as JSON, summary.md and report.md.
A9 [code] The scoring scripts transcript.py, lab_summary.py and validate_runs.py.
A10 [software] EntryLib's scoring rubric, entrylib-rubric 0.2.0 (src/rubric/rubric.json holds the costs; src/rubric/score.py applies them).

HOW TO CITE
This claim: Svetunkov, I. (guarantor) (2026). Claim C9, in: Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study. EntryLib. entrylib:2026.000005@v2#C9. https://entrylib.org/p/2026.000005/C9
The claimset it is from: Svetunkov, I. (guarantor) (2026). Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study. EntryLib. entrylib:2026.000005@v2. https://entrylib.org/p/2026.000005@v2