CLAIMSET Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study
summary Can smaller AI models run a research project with a Claimset lab record and write it up as a claimset? Two Claude Sonnet and two Claude Haiku agents each did one simulation study, comparing how well two Python packages estimate exponential smoothing parameters; four more agents completed a claimset from one of the resulting records. All four lab records passed the tool's own verification and all four claimsets were valid. The Sonnet agents planned their tests before seeing data; the Haiku agents did not. No claim was wrong, but no claimset stated the study's main findings: the lab format turns only pass-or-fail hypotheses into claims, so the agents tested thresholds they chose themselves instead of reporting estimates. EntryLib's scoring rubric ranked a claimset without planned tests, and one without limitations or scope, above more careful ones. These are exploratory results from two runs per model on one task.
licence CC-BY-4.0; contributions C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11; highlights C8, C10, C11, C3, C7
declarations funding: none | competing interests: The guarantor owns EntryLib, which develops the claimset tools, skills and rubric the study evaluates; the study was done to improve them.

DEFINITIONS
D1 [method] claimset-lab: The claimset-lab skill (`.claude/skills/claimset-lab/SKILL.md`, installed by `claimset init --agent claude`) and the `claimset lab` commands of claimset 0.41.6, which keep a lab record (lab record specification v0.21): an append-only, hash-chained log of questions, hypotheses, plans, approvals, runs, results and verdicts, kept on a study branch and pushed after every command.
D2 [method] claimset-submit: The claimset-submit skill of claimset 0.41.6 (claimset specification v0.41), together with `claimset lab export`, `claimset validate` and `claimset render`: it turns a lab record into a claimset and completes it for submission.
D3 [method] Sonnet agent: A Claude Code 2.1.322 subagent started with the Agent tool (type general-purpose) on the model claude-sonnet-5-5, working in a folder of its own that holds the study brief, `AGENTS.md` and the claimset skills as files; it asks the human by ending its turn with its questions. [part of D5]
D4 [method] Haiku agent: The same as the [[D3:Sonnet agent]], on the model claude-haiku-5-5. [part of D5]
D5 [method] agents under test: The two agent configurations compared: the [[D3:Sonnet agent]] and the [[D4:Haiku agent]]. [members D3, D4]
D6 [setting] ETS estimation brief: The research task given to every agent (`study/brief.md`): compare how well smooth 1.2.0 (`ES`) and statsforecast 2.1.1 (`AutoETS` with a fixed model form) estimate the smoothing parameters of the ETS model that generated simulated data, by bias, RMSE and MAE and by whether the RMSE falls as the series length grows; five processes (ETS(A,N,N) with α 0.1 and 0.5, ETS(A,A,N) with (α, β) (0.3, 0.05) and (0.5, 0.2), ETS(A,N,A) with m = 12 and (α, γ) (0.2, 0.1)), series lengths 50, 100, 250 and 500, 300 replicates per cell, smooth's default backcasting as a separate arm, failures counted.
D7 [design] lab phase: Phase 1: four runs, two of the [[D3:Sonnet agent]] and two of the [[D4:Haiku agent]], each in its own git repository with a local remote, given the same task ("Do the study in brief.md with claimset-lab, following its skill, from the first question to the finished record. Ask the human anything you need through me."). The orchestrating agent relays questions and answers them only from [[D9:the fixed answers]], gives no other help, and never shows an agent another run or [[D10:the answer key]]. Runs in parallel on one 4-core machine.
D8 [design] claimset phase: Phase 2: the lab record of run lab-sonnet-1 (the only one that verifies, planned before its data and fixed statsforecast's model form) copied to four folders, with the draft its agent had exported removed; two [[D3:Sonnet agent]] and two [[D4:Haiku agent]] runs, each told "Export this lab record with `claimset lab export` and complete the claimset for submission, following the claimset-submit skill. Do not submit it." Questions answered as in [[D7:the lab phase]].
D9 [design] fixed answers: The answers the orchestrating agent gives in place of the human, the same to every run (`study/human.md`): name and ORCID iD; licence CC-BY-4.0; funding none; the competing interest of the brief; "Approved." for a plan that names its quantities and decision rule before the data are run, otherwise "Please state the quantities and the decision rule first."; for highlights "Choose up to three principal claims you consider most important."; for anything else "Use your judgement and say what you decided." [part of D7, D8]
D10 [data] answer key: The findings a correct paper on [[D6:the brief]] would state, with their numbers (`reference/answer_key.md`), from a reference simulation of the full design written by the orchestrating agent before Phase 1 without claimset-lab (`reference/simulate.py`, `optcheck.py`, `damped_check.py`). Amended once during Phase 1 (statsforecast's β cap).
D11 [variable] record verifies: Whether `claimset lab verify` reports the lab record OK at the end of a run (hash chain, rules, run files).
D12 [variable] tool refusals: Number of commands that `claimset` refused in a run with a lab rule (L1–L30) or an argument or overwrite error; validator errors on a claimset are counted separately.
D13 [variable] plans before data: Whether each confirmatory run of a lab record ran under a plan with its decision rule recorded, and approved by the human, before the run, and whether its hypotheses were written before any run informed them: `all`, `all, hypotheses after pilots` or `none` (no plan).
D14 [variable] model form fixed: Whether a run's simulation script fixed statsforecast's model form as the brief asks (`damped=False`); without it, `AutoETS(model="AAN")` and `ets_f(..., model="AAN")` choose between ETS(A,A,N) and ETS(A,Ad,N).
D15 [variable] convergence checked: Whether a run's script read any optimiser status (for statsforecast, the iteration count `model_["fit"].nit`) and counted fits that did not converge.
D16 [variable] output tokens: Model output tokens of an agent over its whole run, summed from the usage of every model call in its transcript. [unit 1]
D17 [variable] active time: Time an agent spent working in a run, excluding waits for answers, as reported by the harness. [unit min]
D18 [variable] record anchored: Whether the agent anchored the finished record (`claimset lab anchor`).
D19 [variable] runs written as the human: Number of run events an agent recorded with `--by` the human's actor ID.
D20 [variable] claimset validates: Whether `claimset validate` reports the final claimset valid (no errors).
D21 [variable] validator outcomes until valid: The position, among the validator outcomes a claimset run saw (the export's own summary included), of the first with no errors. [unit 1]
D22 [variable] warnings left: Validator warnings on the final claimset. [unit 1]
D23 [method] EntryLib rubric 0.2: The scoring rubric EntryLib applies to published claimsets (entrylib-rubric 0.2.0), run on this machine with `claimset score` and without `--novelty`: each principal claim starts at 100 and loses fixed costs for its problems (support, plan, checkability, context, open limitations and assumptions); the paper's score is that of its lowest principal claim.
D24 [variable] reliability score: The paper score of a claimset under [[D23:EntryLib rubric 0.2]], 0–100. [unit 1]
D25 [variable] claim correctness: Each claim of a claimset checked against [[D10:the answer key]]: `correct` (direction and magnitude agree within Monte Carlo error), `partly correct` (direction agrees; magnitude, scope or a qualifier does not), `wrong`, or `not supported by the record`.
D26 [variable] answer-key findings stated: Number of the answer key's findings on bias (K2–K5), statsforecast's non-convergence (K1.2) and statsforecast's β cap (K8) that a claimset states as claims. [unit 1]
D27 [variable] principal claims with scope: Number of a claimset's principal claims that have a scope. [unit 1]
D28 [variable] limitations: Number of limitations a claimset records. [unit 1]
D29 [variable] research agent's model kept: Whether a completed claimset keeps the model of the agent that did the research, as the lab record names it.
D30 [variable] interventions: Number of messages the orchestrating agent sent a run that were not answers from [[D9:the fixed answers]]. [unit 1]
D31 [design] reference simulation: The reference analysis behind [[D10:the answer key]]: 300 series per cell of [[D6:the brief]], each fitted by smooth with optimised initial states, smooth with backcasting and statsforecast `AutoETS(model, season_length, damped=False)` with its defaults (likelihood, bounds `both`), seeds `numpy.random.default_rng([20261111, process, T, replicate])`.
D32 [concept] statsforecast β cap: In statsforecast 2.1.1 (`statsforecast.ets.initparam`), when α is estimated its starting value is $$\alpha_0 = 0.0001 + 0.2 \cdot \frac{0.9999 - 0.0001}{m}$$ (0.20006 for $m = 1$), and the upper bound of β is set to $\min(0.9999, \alpha_0)$ for the whole optimisation, so β cannot exceed $\alpha_0$.
D33 [variable] decision thresholds: The thresholds a lab record's confirmatory plans fix in their decision rules, as the plans state them; `none` when a record has no plan.
D34 [variable] hypotheses about an estimate: Number of a lab record's hypotheses that state the direction or size of an estimate (a bias towards zero, an estimate on a bound, a bias of about −0.04) rather than a pass-or-fail threshold. [unit 1]
D35 [variable] rubric cost: Points that [[D23:EntryLib rubric 0.2]] deducts from a claim's score for one problem, as its file rubric.json states them. [unit 1]
D36 [design] reading of the rubric's code: The orchestrating agent read the costs in src/rubric/rubric.json and the rule that applies the plan check in src/rubric/score.py (lines 28 and 111), at the commit artefact A10 names.
D37 [variable] statsforecast iteration cap: Number of statsforecast fits whose optimiser stopped at its iteration cap (`nit` = 1001) in [[D31:the reference simulation]]. [unit 1]

CLAIMS
C1 [descriptive; principal; support: empirical_quant E1 (E1.T1) artefacts A1; assumes C21; limitations L2, L5; strength moderate (author); verifiability open; planning exploratory]
  Every lab record that the [[D5:agents under test]] kept with [[D1:claimset-lab]] on [[D6:the ETS estimation brief]] passes `claimset lab verify` ([[D11:record verifies]]).
  scope: data_type: agent runs on one simulation study task | conditions: claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; records as left at the end of each lab run | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
  addresses B6: Agents that run research end to end, judged by an automated reviewer; here agents keep a verifiable lab record of one study, checked by the record's own verification.
C2 [descriptive; principal; support: empirical_quant E1 (E1.T1.r1, E1.T1.r2) artefacts A1, A2; assumes C22; limitations L1, L2, L5; strength low (author); verifiability open; planning exploratory]
  In both of its lab runs, the [[D3:Sonnet agent]] ran every confirmatory test under a plan with a computed decision rule, written with its hypotheses before any run and approved by the human before the test ([[D13:plans before data]]).
  scope: data_type: agent runs on one simulation study task | conditions: claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; the two Sonnet lab runs | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
  addresses B1: Argues that predictions must be fixed before the data are seen; here the question is whether AI agents do so when a tool asks them to.
C3 [descriptive; principal; support: empirical_quant E1 (E1.T1.r3, E1.T1.r4) artefacts A1, A2; assumes C22; limitations L1, L2, L5; strength low (author); verifiability open; planning exploratory]
  Neither lab run of the [[D4:Haiku agent]] planned its tests before seeing the data ([[D13:plans before data]]): one run made no plan at all, and the other set its hypotheses and thresholds after three pilot runs.
  scope: data_type: agent runs on one simulation study task | conditions: claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; the two Haiku lab runs | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
  addresses B1: Argues that predictions must be fixed before the data are seen; here AI agents with the same tool did not, in either of two runs.
  addresses B2: Mandatory registration of trials was followed by more null results; here an agent that planned nothing reported more descriptive findings, on one task.
C4 [descriptive; principal; support: empirical_quant E1 (E1.T1) artefacts A1; limitations L1, L5; strength low (author); verifiability open; planning exploratory]
  Three of the four lab runs of the [[D5:agents under test]] left statsforecast's damped-trend option unset, so that their statsforecast arm selected between a damped and an undamped trend, against the fixed model forms of [[D6:the brief]] ([[D14:model form fixed]]).
  scope: data_type: agent runs on one simulation study task | conditions: claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; the statsforecast arm of each run's script | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
C5 [descriptive; principal; support: empirical_quant E2 (E2.T1) artefacts A1, A2; assumes C22; limitations L2, L5, L6; strength moderate (author); verifiability open; planning exploratory]
  Every claimset that the [[D5:agents under test]] completed with [[D2:claimset-submit]] from one lab record validates with no errors ([[D20:claimset validates]]), at the fourth to sixth validator outcome ([[D21:validator outcomes until valid]]).
  scope: data_type: agent runs on one simulation study task | conditions: one lab record (lab-sonnet-1) completed four times; claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; the final file of each claimset run | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
  addresses B6: Agents write papers judged by an automated reviewer; here agents complete a claimset judged by a validator and against an answer key.
C6 [descriptive; principal; support: empirical_quant E2 (E2.T1) artefacts A6, A8; assumes C21; limitations L3, L4, L5; strength low (author); verifiability open; planning exploratory]
  No claim of the four claimsets that the [[D5:agents under test]] completed with [[D2:claimset-submit]] is wrong against [[D10:the answer key]]: each is correct or partly correct ([[D25:claim correctness]]).
  scope: data_type: agent runs on one simulation study task | conditions: one lab record (lab-sonnet-1) completed four times; claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; the 20 claims of the four claimsets | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
  addresses B6: Agent-written papers judged by an automated reviewer; here each claim is checked against an answer key from a reference simulation.
C7 [descriptive; principal; support: empirical_quant E2 (E2.T1) artefacts A6, A8; assumes C21; uses C12; limitations L3, L4, L5; strength low (author); verifiability open; planning exploratory]
  None of the four claimsets that the [[D5:agents under test]] completed with [[D2:claimset-submit]] states as a claim any of the main findings of [[D10:the answer key]]: how large the estimation bias is, that statsforecast's optimiser fails to converge for the seasonal model, or that statsforecast caps β ([[D26:answer-key findings stated]]).
  scope: data_type: agent runs on one simulation study task | conditions: one lab record (lab-sonnet-1) completed four times; claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; claims only, not limitations or open questions | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
  addresses B6: Agent-written papers judged by an automated reviewer; here the check is which findings of a reference analysis the claims state.
C8 [interpretive; principal; support: empirical_quant E1 (E1.T2) artefacts A1, argument W1; uses C7, C12; limitations L1, L5; strength low (author); verifiability open; planning exploratory]
  The [[D5:agents under test]] answered [[D6:the brief]]'s questions about how large the bias and RMSE are with pass-or-fail tests against thresholds they chose themselves, so the estimates never became claims: [[D1:claimset-lab]] turns only hypotheses decided by a rule into claims.
  scope: data_type: agent runs on one simulation study task | conditions: claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; the three lab runs with a plan and the one without | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
  addresses B4: Recommends framing research questions as estimation rather than significance tests; here a tool's pass-or-fail plans led agents away from estimation.
  addresses B3: Registered Reports report far fewer positive results than standard articles; here the registered format changed what agents reported, from estimates to threshold tests.
C9 [descriptive; principal; support: empirical_quant E2 (E2.T1) artefacts A1; limitations L1, L2, L5; strength low (author); verifiability open; planning exploratory]
  The two claimsets that the [[D4:Haiku agent]] completed give a scope to none of their three principal claims, against all three in each of the two that the [[D3:Sonnet agent]] completed ([[D27:principal claims with scope]]).
  scope: data_type: agent runs on one simulation study task | conditions: one lab record (lab-sonnet-1) completed four times; claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; principal claims only | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
C10 [descriptive; principal; support: empirical_quant E2 (E2.T1, E2.T2) artefacts A8, A10; uses C14; limitations L5; strength moderate (author); verifiability open; planning exploratory]
  Under [[D23:EntryLib rubric 0.2]], the one claimset that tested nothing under a plan gets a higher [[D24:reliability score]] than each of the six claimsets whose principal claims were planned tests.
  scope: conditions: entrylib-rubric 0.2.0, without --novelty; the six claimsets of this study; claimsets written in either phase | excludes: other rubric versions
  addresses B5: Incentives that reward output select for weaker methods; here a scoring rubric ranks a claimset without planned tests above planned ones.
C11 [descriptive; principal; support: empirical_quant E2 (E2.T1.r1, E2.T1.r4) artefacts A8, A10; uses C13; limitations L5; strength moderate (author); verifiability open; planning exploratory]
  Under [[D23:EntryLib rubric 0.2]], a claimset that states no limitations and gives its claims no scope (claimset-haiku-2) gets a higher [[D24:reliability score]] than one written from the same lab record with scoped claims and three specific limitations (claimset-sonnet-1).
  scope: conditions: entrylib-rubric 0.2.0, without --novelty; the six claimsets of this study; claimsets completed from one lab record | excludes: other rubric versions
  addresses B5: Incentives that reward output select for weaker methods; here a scoring rubric ranks a claimset without limitations or scope above one with both.
C12 [descriptive; supporting; not_applicable; support: empirical_quant E2 artefacts A1; strength high (author); verifiability open; planning exploratory]
  The lab record that [[D8:the claimset phase]] started from holds three hypotheses, each a threshold test on the RMSE (a band for the ratio of the two packages, backcasting better in most cells at the shortest length, the longest length against the shortest), and no hypothesis about the size of the bias or the RMSE.
  scope: conditions: the lab record of run lab-sonnet-1, 1,130 events
C13 [descriptive; supporting; not_applicable; support: empirical_quant E4 (E4.T1.r1, E4.T1.r2) artefacts A10; strength high (author); verifiability open; planning exploratory]
  In [[D23:EntryLib rubric 0.2]], an unmitigated limitation that could bias a claim's magnitude costs the claim 12 points, more than a missing scope (10 points).
  scope: conditions: entrylib-rubric 0.2.0, without --novelty; the six claimsets of this study; costs per claim | excludes: other rubric versions
C14 [descriptive; supporting; not_applicable; support: empirical_quant E4 (E4.T1.r3, E4.T1.r4) artefacts A10; strength high (author); verifiability open; planning exploratory]
  In [[D23:EntryLib rubric 0.2]], a descriptive claim made without a plan costs nothing in the plan group, and a planned claim whose plan is anchored only by a git push costs 3 points.
  scope: conditions: entrylib-rubric 0.2.0, without --novelty; the six claimsets of this study; claims of type descriptive | excludes: other rubric versions
C21 [descriptive; supporting; assumption (methodological, unchecked); verifiability not_applicable]
  [[D10:The answer key]] states the findings that a correct paper on [[D6:the brief]] would state, with correct numbers.
C22 [descriptive; supporting; assumption (methodological, unchecked); verifiability not_applicable]
  The [[D5:agents under test]], run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders.
C15 [descriptive; secondary; support: empirical_quant E1 (E1.T1) artefacts A1; uses C16; limitations L5; strength low (author); verifiability open; planning exploratory]
  None of the four lab runs of the [[D5:agents under test]] checked whether statsforecast's optimiser converged ([[D15:convergence checked]]).
  scope: data_type: agent runs on one simulation study task | conditions: claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; the statsforecast arm of each run's script | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
C16 [descriptive; secondary; not_applicable; support: simulation E3 (E3.T2) artefacts A6, A7; strength moderate (author); verifiability open; planning exploratory]
  In [[D31:the reference simulation]], statsforecast's optimiser stopped at its iteration cap in nearly every ETS(A,N,A) fit, at every series length ([[D37:statsforecast iteration cap]]).
  scope: conditions: statsforecast 2.1.1 AutoETS defaults; ETS(A,N,A), m = 12, α = 0.2, γ = 0.1
C17 [descriptive; secondary; support: empirical_quant E1 (E1.T1.r3) artefacts A1, A2; limitations L1; strength low (author); verifiability open; planning exploratory]
  In one of its two lab runs, the [[D4:Haiku agent]] recorded a run with the human's actor ID as its author, and `claimset lab` accepted it ([[D19:runs written as the human]]).
  scope: data_type: agent runs on one simulation study task | conditions: claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; the two Haiku lab runs | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
C18 [descriptive; secondary; support: empirical_quant E2 (E2.T1.r3) artefacts A1; limitations L1; strength low (author); verifiability open; planning exploratory]
  In one of its two claimset runs, the [[D4:Haiku agent]] replaced the model of the agent that had done the research, as the lab record names it, with its own ([[D29:research agent's model kept]]).
  scope: data_type: agent runs on one simulation study task | conditions: one lab record (lab-sonnet-1) completed four times; claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; the two Haiku claimset runs | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields
C19 [descriptive; secondary; support: simulation E3 (E3.T1) artefacts A6, A7; strength high (author); verifiability open; planning exploratory]
  When statsforecast 2.1.1 fits ETS(A,A,N) with its defaults, its estimate of β never exceeds its starting value of α ([[D32:statsforecast β cap]]), so that no estimate can lie above a true β of 0.2.
  scope: conditions: statsforecast 2.1.1 AutoETS defaults, model AAN, m = 1; ETS(A,A,N) with (α, β) = (0.5, 0.2); series lengths 50 to 500, 300 series each
C20 [descriptive; secondary; not_applicable; support: empirical_quant E1 (E1.T1.r3) artefacts A2, A6; uses C19; limitations L4; strength low (author); verifiability open; planning exploratory]
  The [[D32:statsforecast β cap]] was reported by a [[D4:Haiku agent]] lab run (lab-haiku-1, in an exploratory run) and was missing from [[D10:the answer key]] as written before the lab phase.
  scope: data_type: agent runs on one simulation study task | conditions: claimset 0.41.6 with its skills read as files; Claude Code 2.1.322 Agent-tool subagents; two runs per agent; fixed answers in place of the human; the lab phase | excludes: agents in sessions of their own with the guard hook active; other study tasks and fields

EVIDENCE
E1 [quant; design experiment; data_origin primary; n 4] [[D7:The lab phase]]: two runs each of the [[D3:Sonnet agent]] and the [[D4:Haiku agent]] did [[D6:the ETS estimation brief]] with [[D1:claimset-lab]]. For each run the orchestrating agent read the final lab record (`tools/lab_summary.py`: plans, approvals and runs in record order, the actor of every event, anchors), the run's script, and its transcript (`tools/transcript.py`: every tool call, every refusal, model-call usage), and recorded the variables of table T1. Every question and answer is in `log/questions.md`.
  setup: studied D3, D4; cases D6; procedure D7, D9, D1; lens D10, D11, D12, D13, D14, D15, D16, D17, D18, D19, D33, D34
  E1.T1 Outcomes of the four lab runs of the lab phase on the ETS estimation brief
  | row | run | agent | D11 | D12 | D13 | D14 | D15 | D18 | D19 | D16 | D17 |
  | r1 | lab-sonnet-1 | D3 | yes | 0 | all | yes | no | yes | 0 | 52609 | 11.2 |
  | r2 | lab-sonnet-2 | D3 | yes | 0 | all | no | no | yes | 0 | 80054 | 13.5 |
  | r3 | lab-haiku-1 | D4 | yes | 3 | none | no | no | yes | 1 | 265066 | 16.7 |
  | r4 | lab-haiku-2 | D4 | yes | 1 | all, hypotheses after pilots | no | no | no | 0 | 122979 | 11.8 |
  E1.T2 The decision thresholds each lab record's plans fixed, and the record's hypotheses about the direction or size of an estimate
  | row | run | agent | D33 | D34 |
  | r1 | lab-sonnet-1 | D3 | RMSE ratio of the two optimised arms between 0.9 and 1.1 in all 32 cells; backcasting with the lower RMSE in at least 6 of 8 cells at length 50; RMSE at length 500 below length 50 in all 24 comparisons | 0 |
  | r2 | lab-sonnet-2 | D3 | RMSE at length 500 below length 50 in all 24 series; smooth with the lower RMSE than statsforecast in at least 75 % of 32 comparisons; backcasting below optimised initial states in at least 75 %; |bias| + 1.645 SE below 0.02 at length 500 for every parameter | 0 |
  | r3 | lab-haiku-1 | D4 | none | 3 |
  | r4 | lab-haiku-2 | D4 | no violations of: RMSE falling with length; statsforecast's RMSE within 10 % of smooth's; |bias| at most 0.02 at length 500 (not supported only at 12, 16 and 12 violations) | 0 |
E2 [quant; design experiment; data_origin primary; n 4] [[D8:The claimset phase]]: two runs each of the [[D3:Sonnet agent]] and the [[D4:Haiku agent]] exported one lab record and completed the claimset with [[D2:claimset-submit]]. For each final claimset the orchestrating agent ran `claimset validate` and `claimset score` ([[D23:EntryLib rubric 0.2]]), counted the validator outcomes in the transcript (`tools/validate_runs.py`), checked that no record-derived field (hypothesis, planning, planned_at, test, exploration counts) had changed, and scored every claim against [[D10:the answer key]] (table T1). Table T2 scores, under the same rubric, the two claimsets that lab-phase agents wrote on their own records after finishing them (beyond their task).
  setup: studied D3, D4; cases D6; procedure D8, D9, D2; lens D10, D23, D20, D21, D22, D24, D25, D26, D27, D28, D29, D30, D16
  E2.T1 Outcomes of the four claimset runs of the claimset phase, all from the lab record of lab-sonnet-1
  | row | run | agent | D20 | D21 | D22 | D24 | D25 {outcome: correct} | D25 {outcome: partly correct} | D25 {outcome: wrong} | D26 | D27 | D28 | D29 | D30 | D16 |
  | r1 | claimset-sonnet-1 | D3 | yes | 4 | 2 | 73 | 4 | 1 | 0 | 0 | 3 | 3 | yes | 0 | 50493 |
  | r2 | claimset-sonnet-2 | D3 | yes | 5 | 3 | 72 | 5 | 0 | 0 | 0 | 3 | 3 | yes | 0 | 44792 |
  | r3 | claimset-haiku-1 | D4 | yes | 6 | 1 | 79 | 4 | 1 | 0 | 0 | 0 | 3 | no | 0 | 69641 |
  | r4 | claimset-haiku-2 | D4 | yes | 4 | 11 | 78 | 3 | 2 | 0 | 0 | 0 | 0 | yes | 1 | 55371 |
  E2.T2 EntryLib rubric 0.2 scores of the two claimsets written by lab-phase agents on their own records
  | row | run | agent | planning of the principal claims | D24 | D28 | D27 |
  | r1 | lab-sonnet-2 | D3 | confirmatory | 73 | 4 | 3 |
  | r2 | lab-haiku-1 | D4 | exploratory | 92 | 0 | 3 |
E3 [simulation; data_origin primary; runs 300] [[D31:The reference simulation]] of [[D6:the brief]] (R = 300 per cell, 20 cells, three arms, 110 s on 4 cores), with `optcheck.py` and `damped_check.py`. Table T1: statsforecast's β estimates for ETS(A,A,N) with (α, β) = (0.5, 0.2), against smooth's; table T2: statsforecast fits of ETS(A,N,A) that stopped at the optimiser's iteration cap. The β cap was first reported by run lab-haiku-1 (its exploratory run RN1); it was verified here against the statsforecast 2.1.1 source and these estimates, and added to the answer key.
  setup: studied D31; cases D6; procedure D31; lens D32, D37
  E3.T1 statsforecast and smooth (optimised initial states) estimates of β for ETS(A,A,N) with α = 0.5, β = 0.2, 300 series per length
  | row | series length | largest statsforecast β estimate | statsforecast fits within 0.001 of 0.20006 | smooth fits with β above 0.20006 |
  | r1 | 50 | 0.20006 | 100 | 131 |
  | r2 | 100 | 0.20006 | 135 | 146 |
  | r3 | 250 | 0.20006 | 132 | 132 |
  | r4 | 500 | 0.20006 | 144 | 137 |
  E3.T2 statsforecast fits of ETS(A,N,A) (α = 0.2, γ = 0.1, m = 12) at the optimiser's iteration cap, of 300 per length
  | row | series length | D37 |
  | r1 | 50 | 288 |
  | r2 | 100 | 296 |
  | r3 | 250 | 300 |
  | r4 | 500 | 299 |
  parameters: A6
  seeds: ["20261111"]
E4 [quant; design other; data_origin secondary] [[D36:A reading of the rubric's code]]: the costs [[D23:EntryLib rubric 0.2]] deducts for the problems that decide the scores compared in this paper.
  setup: studied D23; cases D23; procedure D36; lens D35
  E4.T1 Costs of four problems in EntryLib rubric 0.2, per claim
  | row | problem | D35 |
  | r1 | a limitation that could bias magnitude, not mitigated | 12 |
  | r2 | no scope | 10 |
  | r3 | a plan anchored only by a git push | 3 |
  | r4 | a descriptive claim without a plan | 0 |

WARRANTS
W1 [inductive; premises C12] The claimset-lab skill offers one route from a run to a claim: a hypothesis decided by a rule, with computed rules preferred whenever the outcome can be computed from numbers (SKILL.md, lines 83–100), and export makes claims only from hypotheses with a verdict or a final status (lab record v0.21 §9). Every planned run accordingly chose thresholds the brief did not give (an RMSE ratio within 10 %, |bias| at most 0.02, shares of at least 0.75, at least 12 violations), and its claims are the verdicts on them; the one run without a plan wrote notes and claims about the estimates themselves (the γ pile-up, statsforecast's seasonal α bias, the β cap).

LIMITATIONS
L1 [sample_size; affects C2, C3, C4, C8, C9, C17, C18; could_reverse; mitigation none] Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs.
L2 [confounding; affects C1, C2, C3, C5, C9; could_bias_magnitude; mitigation none] The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine.
L3 [measurement; affects C6, C7; could_bias_magnitude; mitigation partial] One scorer, the orchestrating agent, wrote the answer key and the protocol and scored every claim, knowing which agent wrote it.
L4 [deviation; affects C6, C7, C20; could_bias_magnitude; mitigation none] The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key.
L5 [sampling; affects C1, C2, C3, C4, C15, C5, C6, C7, C8, C9, C10, C11; reduces_transferability; mitigation none] One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322).
L6 [deviation; affects C5; could_bias_magnitude; mitigation none] One claimset run (claimset-haiku-2) stopped with its own stated work undone and was sent one message with no content ("Continue.") before it finished.

OPEN QUESTIONS
O1 [from C2, C3, C9, C20] Whether the differences between the Sonnet and Haiku agents hold over more runs, other study tasks and agents in sessions of their own is not established.
O2 [from C17, C18] Whether refusing writes by a human actor in agent harnesses, and keeping the research agent's model through export, would prevent the misattributions of a run and of a study's research is not tested.
O3 [from C4, C15] Whether a skill instruction to report each fit's model form and optimiser status as results would close the gaps in fixing the model form and checking convergence is not tested.
O4 [from C7, C8] Whether a plan that fixes estimates rather than a decision rule would lead agents to state the brief's findings as claims is not tested.

ARTEFACTS
A1 [data] The eight runs as the agents left them: each folder's files without its git directory, and its full git history (study branch included) in repo.bundle.
A2 [trace] The eight agents' transcripts (Claude Code subagent JSONL), with every tool call and result and the usage of every model call.
A3 [prompt] The task prompts given to the agents in each phase.
A4 [protocol] The study brief and the fixed answers given in place of the human.
A5 [data] Setup notes, every question and answer, agent IDs and usage per turn.
A6 [data] The reference analysis: estimates.csv, summary.csv, consistency.csv, optcheck.csv, damped_check.csv and answer_key.md.
A7 [code] The reference analysis scripts simulate.py, optcheck.py and damped_check.py.
A8 [data] Scoring per run, rubric scores as JSON, summary.md and report.md.
A9 [code] The scoring scripts transcript.py, lab_summary.py and validate_runs.py.
A10 [software] EntryLib's scoring rubric, entrylib-rubric 0.2.0 (src/rubric/rubric.json holds the costs; src/rubric/score.py applies them).

REFERENCES
B1 10.1073/pnas.1708274114 (doi)
B2 10.1371/journal.pone.0132382 (doi)
B3 10.1177/25152459211007467 (doi)
B4 10.1177/0956797613504966 (doi)
B5 10.1098/rsos.160384 (doi)
B6 2408.06292 (arxiv)

SCORES
Scores are computed by rules from the claimset file (EntryLib rubric 0.3): they check the record, not the science, and may change as the rubric and reviews develop.
Reliability of the paper: 76 (medium), the mean of its 11 principal claims; the lowest is C9 at 66.
C1: reliability 75 (medium); novelty: new, by the authors
C2: reliability 78 (medium); novelty: new, by the authors
C3: reliability 78 (medium); novelty: new, by the authors
C4: reliability 69 (medium); novelty: new, by the authors
C5: reliability 72 (medium); novelty: new, by the authors
C6: reliability 74 (medium); novelty: new, by the authors
C7: reliability 74 (medium); novelty: new, by the authors
C8: reliability 67 (medium); novelty: new, by the authors
C9: reliability 66 (medium); novelty: new, by the authors
C10: reliability 84 (high); novelty: new, by the authors
C11: reliability 94 (high); novelty: new, by the authors
C12: reliability 90 (high)
C13: reliability 100 (high)
C14: reliability 100 (high)
C15: reliability 89 (high); novelty: new, by the authors
C16: reliability 90 (high)
C17: reliability 90 (high); novelty: new, by the authors
C18: reliability 90 (high); novelty: new, by the authors
C19: reliability 90 (high); novelty: new, by the authors
C20: reliability 84 (high)
Open problems in the principal claims and what they rest on:
P1 [open; up to -10; C2, C3, C4, C8, C9, C17, C18] Limitation L1 (could reverse, not mitigated): Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P2 [support; up to -10; C1, C4, C5, C6, C7, C8, C9, C10, C12, C15, C16, C19, C20] Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
P3 [plan; up to -5; C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11] A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
P4 [context; up to -5; C4, C9] A principal claim related to no earlier work. To remove it: Relate the claim to the earlier work it extends, addresses or departs from.
P5 [open; up to -6; C1, C2, C3, C5, C9] Limitation L2 (could bias magnitude, not mitigated): The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P6 [open; up to -3; C1, C6, C7, C8] Assumption C21 is not checked: [[D10:The answer key]] states the findings that a correct paper on [[D6:the brief]] would state, with correct numbers. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
P7 [open; up to -1; C6, C7, C8] Limitation L3 (could bias magnitude, mitigation partial): One scorer, the orchestrating agent, wrote the answer key and the protocol and scored every claim, knowing which agent wrote it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P8 [open; up to -6; C6, C7, C8, C20] Limitation L4 (could bias magnitude, not mitigated): The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P9 [open; up to -1; C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C15] Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P10 [open; up to -3; C5] Limitation L6 (could bias magnitude, not mitigated): One claimset run (claimset-haiku-2) stopped with its own stated work undone and was sent one message with no content ("Continue.") before it finished. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
P11 [open; up to -3; C2, C3, C5] Assumption C22 is not checked: The [[D5:agents under test]], run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders. To remove it: Test the assumption, or analyse how sensitive the claim is to it.

HOW TO CITE
This claimset: Svetunkov, I. (guarantor) (2026). Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study. EntryLib. entrylib:2026.000005@v2. https://entrylib.org/p/2026.000005@v2
One claim of it: by its identifier, as entrylib:2026.000005@v2#C1 for claim C1. Each principal claim has its own page, as https://entrylib.org/p/2026.000005/C1.