CLAIM C11 of entrylib:2026.000005 Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study
licence CC-BY-4.0; contributions C11

DEFINITIONS
D2 [method] claimset-submit: The claimset-submit skill of claimset 0.41.6 (claimset specification v0.41), together with `claimset lab export`, `claimset validate` and `claimset render`: it turns a lab record into a claimset and completes it for submission.
D3 [method] Sonnet agent: A Claude Code 2.1.322 subagent started with the Agent tool (type general-purpose) on the model claude-sonnet-5-5, working in a folder of its own that holds the study brief, `AGENTS.md` and the claimset skills as files; it asks the human by ending its turn with its questions. [part of D5]
D4 [method] Haiku agent: The same as the [[D3:Sonnet agent]], on the model claude-haiku-5-5. [part of D5]
D5 [method] agents under test: The two agent configurations compared: the [[D3:Sonnet agent]] and the [[D4:Haiku agent]]. [members D3, D4]
D6 [setting] ETS estimation brief: The research task given to every agent (`study/brief.md`): compare how well smooth 1.2.0 (`ES`) and statsforecast 2.1.1 (`AutoETS` with a fixed model form) estimate the smoothing parameters of the ETS model that generated simulated data, by bias, RMSE and MAE and by whether the RMSE falls as the series length grows; five processes (ETS(A,N,N) with α 0.1 and 0.5, ETS(A,A,N) with (α, β) (0.3, 0.05) and (0.5, 0.2), ETS(A,N,A) with m = 12 and (α, γ) (0.2, 0.1)), series lengths 50, 100, 250 and 500, 300 replicates per cell, smooth's default backcasting as a separate arm, failures counted.
D7 [design] lab phase: Phase 1: four runs, two of the [[D3:Sonnet agent]] and two of the [[D4:Haiku agent]], each in its own git repository with a local remote, given the same task ("Do the study in brief.md with claimset-lab, following its skill, from the first question to the finished record. Ask the human anything you need through me."). The orchestrating agent relays questions and answers them only from [[D9:the fixed answers]], gives no other help, and never shows an agent another run or [[D10:the answer key]]. Runs in parallel on one 4-core machine.
D8 [design] claimset phase: Phase 2: the lab record of run lab-sonnet-1 (the only one that verifies, planned before its data and fixed statsforecast's model form) copied to four folders, with the draft its agent had exported removed; two [[D3:Sonnet agent]] and two [[D4:Haiku agent]] runs, each told "Export this lab record with `claimset lab export` and complete the claimset for submission, following the claimset-submit skill. Do not submit it." Questions answered as in [[D7:the lab phase]].
D9 [design] fixed answers: The answers the orchestrating agent gives in place of the human, the same to every run (`study/human.md`): name and ORCID iD; licence CC-BY-4.0; funding none; the competing interest of the brief; "Approved." for a plan that names its quantities and decision rule before the data are run, otherwise "Please state the quantities and the decision rule first."; for highlights "Choose up to three principal claims you consider most important."; for anything else "Use your judgement and say what you decided." [part of D7, D8]
D10 [data] answer key: The findings a correct paper on [[D6:the brief]] would state, with their numbers (`reference/answer_key.md`), from a reference simulation of the full design written by the orchestrating agent before Phase 1 without claimset-lab (`reference/simulate.py`, `optcheck.py`, `damped_check.py`). Amended once during Phase 1 (statsforecast's β cap).
D16 [variable] output tokens: Model output tokens of an agent over its whole run, summed from the usage of every model call in its transcript. [unit 1]
D20 [variable] claimset validates: Whether `claimset validate` reports the final claimset valid (no errors).
D21 [variable] validator outcomes until valid: The position, among the validator outcomes a claimset run saw (the export's own summary included), of the first with no errors. [unit 1]
D22 [variable] warnings left: Validator warnings on the final claimset. [unit 1]
D23 [method] EntryLib rubric 0.2: The scoring rubric EntryLib applies to published claimsets (entrylib-rubric 0.2.0), run on this machine with `claimset score` and without `--novelty`: each principal claim starts at 100 and loses fixed costs for its problems (support, plan, checkability, context, open limitations and assumptions); the paper's score is that of its lowest principal claim.
D24 [variable] reliability score: The paper score of a claimset under [[D23:EntryLib rubric 0.2]], 0–100. [unit 1]
D25 [variable] claim correctness: Each claim of a claimset checked against [[D10:the answer key]]: `correct` (direction and magnitude agree within Monte Carlo error), `partly correct` (direction agrees; magnitude, scope or a qualifier does not), `wrong`, or `not supported by the record`.
D26 [variable] answer-key findings stated: Number of the answer key's findings on bias (K2–K5), statsforecast's non-convergence (K1.2) and statsforecast's β cap (K8) that a claimset states as claims. [unit 1]
D27 [variable] principal claims with scope: Number of a claimset's principal claims that have a scope. [unit 1]
D28 [variable] limitations: Number of limitations a claimset records. [unit 1]
D29 [variable] research agent's model kept: Whether a completed claimset keeps the model of the agent that did the research, as the lab record names it.
D30 [variable] interventions: Number of messages the orchestrating agent sent a run that were not answers from [[D9:the fixed answers]]. [unit 1]
D35 [variable] rubric cost: Points that [[D23:EntryLib rubric 0.2]] deducts from a claim's score for one problem, as its file rubric.json states them. [unit 1]
D36 [design] reading of the rubric's code: The orchestrating agent read the costs in src/rubric/rubric.json and the rule that applies the plan check in src/rubric/score.py (lines 28 and 111), at the commit artefact A10 names.

CLAIMS
C11 [descriptive; principal; support: empirical_quant E2 (E2.T1.r1, E2.T1.r4) artefacts A8, A10; uses C13; limitations L5; strength moderate (author); verifiability open; planning exploratory]
  Under [[D23:EntryLib rubric 0.2]], a claimset that states no limitations and gives its claims no scope (claimset-haiku-2) gets a higher [[D24:reliability score]] than one written from the same lab record with scoped claims and three specific limitations (claimset-sonnet-1).
  scope: conditions: entrylib-rubric 0.2.0, without --novelty; the six claimsets of this study; claimsets completed from one lab record | excludes: other rubric versions
  addresses B5: Incentives that reward output select for weaker methods; here a scoring rubric ranks a claimset without limitations or scope above one with both.
C13 [descriptive; supporting; not_applicable; support: empirical_quant E4 (E4.T1.r1, E4.T1.r2) artefacts A10; strength high (author); verifiability open; planning exploratory]
  In [[D23:EntryLib rubric 0.2]], an unmitigated limitation that could bias a claim's magnitude costs the claim 12 points, more than a missing scope (10 points).
  scope: conditions: entrylib-rubric 0.2.0, without --novelty; the six claimsets of this study; costs per claim | excludes: other rubric versions

EVIDENCE
E2 [quant; design experiment; data_origin primary; n 4] [[D8:The claimset phase]]: two runs each of the [[D3:Sonnet agent]] and the [[D4:Haiku agent]] exported one lab record and completed the claimset with [[D2:claimset-submit]]. For each final claimset the orchestrating agent ran `claimset validate` and `claimset score` ([[D23:EntryLib rubric 0.2]]), counted the validator outcomes in the transcript (`tools/validate_runs.py`), checked that no record-derived field (hypothesis, planning, planned_at, test, exploration counts) had changed, and scored every claim against [[D10:the answer key]] (table T1). Table T2 scores, under the same rubric, the two claimsets that lab-phase agents wrote on their own records after finishing them (beyond their task).
  setup: studied D3, D4; cases D6; procedure D8, D9, D2; lens D10, D23, D20, D21, D22, D24, D25, D26, D27, D28, D29, D30, D16
  E2.T1 Outcomes of the four claimset runs of the claimset phase, all from the lab record of lab-sonnet-1
  | row | run | agent | D20 | D21 | D22 | D24 | D25 {outcome: correct} | D25 {outcome: partly correct} | D25 {outcome: wrong} | D26 | D27 | D28 | D29 | D30 | D16 |
  | r1 | claimset-sonnet-1 | D3 | yes | 4 | 2 | 73 | 4 | 1 | 0 | 0 | 3 | 3 | yes | 0 | 50493 |
  | r2 | claimset-sonnet-2 | D3 | yes | 5 | 3 | 72 | 5 | 0 | 0 | 0 | 3 | 3 | yes | 0 | 44792 |
  | r3 | claimset-haiku-1 | D4 | yes | 6 | 1 | 79 | 4 | 1 | 0 | 0 | 0 | 3 | no | 0 | 69641 |
  | r4 | claimset-haiku-2 | D4 | yes | 4 | 11 | 78 | 3 | 2 | 0 | 0 | 0 | 0 | yes | 1 | 55371 |
  E2.T2 EntryLib rubric 0.2 scores of the two claimsets written by lab-phase agents on their own records
  | row | run | agent | planning of the principal claims | D24 | D28 | D27 |
  | r1 | lab-sonnet-2 | D3 | confirmatory | 73 | 4 | 3 |
  | r2 | lab-haiku-1 | D4 | exploratory | 92 | 0 | 3 |
E4 [quant; design other; data_origin secondary] [[D36:A reading of the rubric's code]]: the costs [[D23:EntryLib rubric 0.2]] deducts for the problems that decide the scores compared in this paper.
  setup: studied D23; cases D23; procedure D36; lens D35
  E4.T1 Costs of four problems in EntryLib rubric 0.2, per claim
  | row | problem | D35 |
  | r1 | a limitation that could bias magnitude, not mitigated | 12 |
  | r2 | no scope | 10 |
  | r3 | a plan anchored only by a git push | 3 |
  | r4 | a descriptive claim without a plan | 0 |

LIMITATIONS
L5 [sampling; affects C1, C2, C3, C4, C15, C5, C6, C7, C8, C9, C10, C11; reduces_transferability; mitigation none] One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322).

ARTEFACTS
A1 [data] The eight runs as the agents left them: each folder's files without its git directory, and its full git history (study branch included) in repo.bundle.
A2 [trace] The eight agents' transcripts (Claude Code subagent JSONL), with every tool call and result and the usage of every model call.
A3 [prompt] The task prompts given to the agents in each phase.
A6 [data] The reference analysis: estimates.csv, summary.csv, consistency.csv, optcheck.csv, damped_check.csv and answer_key.md.
A8 [data] Scoring per run, rubric scores as JSON, summary.md and report.md.
A9 [code] The scoring scripts transcript.py, lab_summary.py and validate_runs.py.
A10 [software] EntryLib's scoring rubric, entrylib-rubric 0.2.0 (src/rubric/rubric.json holds the costs; src/rubric/score.py applies them).

HOW TO CITE
This claim: Svetunkov, I. (guarantor) (2026). Claim C11, in: Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study. EntryLib. entrylib:2026.000005@v2#C11. https://entrylib.org/p/2026.000005/C11
The claimset it is from: Svetunkov, I. (guarantor) (2026). Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study. EntryLib. entrylib:2026.000005@v2. https://entrylib.org/p/2026.000005@v2