The claim
In both of its lab runs, the Sonnet agent ran every confirmatory test under a plan with a computed decision rule, written with its hypotheses before any run and approved by the human before the test (plans before data).
Scope: data: agent runs on one simulation study task; conditions: claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the two Sonnet lab runs; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E1 (T1.r1, T1.r2), artefacts A1, A2 · Assumes: C22 · Limitations: L1, L2, L5 · Addresses: B1
Evidence
The lab phase: two runs each of the Sonnet agent and the Haiku agent did the ETS estimation brief with claimset-lab. For each run the orchestrating agent read the final lab record (tools/lab_summary.py: plans, approvals and runs in record order, the actor of every event, anchors), the run's script, and its transcript (tools/transcript.py: every tool call, every refusal, model-call usage), and recorded the variables of table T1. Every question and answer is in log/questions.md.
Setup: studied Sonnet agent, Haiku agent · cases ETS estimation brief · procedure lab phase, fixed answers, claimset-lab · lens answer key, record verifies, tool refusals, plans before data, model form fixed, convergence checked, output tokens, active time, record anchored, runs written as the human, decision thresholds, hypotheses about an estimate
| Row | run | agent | record verifies | tool refusals | plans before data | model form fixed | convergence checked | record anchored | runs written as the human | output tokens | active time |
|---|---|---|---|---|---|---|---|---|---|---|---|
| r1 | lab-sonnet-1 | Sonnet agent | yes | 0 | all | yes | no | yes | 0 | 52609 | 11.2 |
| r2 | lab-sonnet-2 | Sonnet agent | yes | 0 | all | no | no | yes | 0 | 80054 | 13.5 |
| r3 | lab-haiku-1 | Haiku agent | yes | 3 | none | no | no | yes | 1 | 265066 | 16.7 |
| r4 | lab-haiku-2 | Haiku agent | yes | 1 | all, hypotheses after pilots | no | no | no | 0 | 122979 | 11.8 |
| Row | run | agent | decision thresholds | hypotheses about an estimate |
|---|---|---|---|---|
| r1 | lab-sonnet-1 | Sonnet agent | RMSE ratio of the two optimised arms between 0.9 and 1.1 in all 32 cells; backcasting with the lower RMSE in at least 6 of 8 cells at length 50; RMSE at length 500 below length 50 in all 24 comparisons | 0 |
| r2 | lab-sonnet-2 | Sonnet agent | RMSE at length 500 below length 50 in all 24 series; smooth with the lower RMSE than statsforecast in at least 75 % of 32 comparisons; backcasting below optimised initial states in at least 75 %; |bias| + 1.645 SE below 0.02 at length 500 for every parameter | 0 |
| r3 | lab-haiku-1 | Haiku agent | none | 3 |
| r4 | lab-haiku-2 | Haiku agent | no violations of: RMSE falling with length; statsforecast's RMSE within 10 % of smooth's; |bias| at most 0.02 at length 500 (not supported only at 12, 16 and 12 violations) | 0 |
Limitations
Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs.
The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine.
One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322).
Assumptions
The agents under test, run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders. Rested on by C2.
Definitions
Methods 4
D1 claimset-lab: The claimset-lab skill (.claude/skills/claimset-lab/SKILL.md, installed by claimset init --agent claude) and the claimset lab commands of claimset 0.41.6, which keep a lab record (lab record specification v0.21): an append-only, hash-chained log of questions, hypotheses, plans, approvals, runs, results and verdicts, kept on a study branch and pushed after every command.
D5 agents under test: The two agent configurations compared: the Sonnet agent and the Haiku agent. Members: D3, D4.
D3 · part of D5 Sonnet agent: A Claude Code 2.1.322 subagent started with the Agent tool (type general-purpose) on the model claude-sonnet-5-5, working in a folder of its own that holds the study brief, AGENTS.md and the claimset skills as files; it asks the human by ending its turn with its questions.
D4 · part of D5 Haiku agent: The same as the Sonnet agent, on the model claude-haiku-5-5.
Data 1
D10 answer key: The findings a correct paper on the brief would state, with their numbers (reference/answer_key.md), from a reference simulation of the full design written by the orchestrating agent before Phase 1 without claimset-lab (reference/simulate.py, optcheck.py, damped_check.py). Amended once during Phase 1 (statsforecast's β cap).
Variables and measures 11
D11 record verifies: Whether claimset lab verify reports the lab record OK at the end of a run (hash chain, rules, run files).
D12 tool refusals: Number of commands that claimset refused in a run with a lab rule (L1–L30) or an argument or overwrite error; validator errors on a claimset are counted separately.
D13 plans before data: Whether each confirmatory run of a lab record ran under a plan with its decision rule recorded, and approved by the human, before the run, and whether its hypotheses were written before any run informed them: all, all, hypotheses after pilots or none (no plan).
D14 model form fixed: Whether a run's simulation script fixed statsforecast's model form as the brief asks (damped=False); without it, AutoETS(model="AAN") and ets_f(..., model="AAN") choose between ETS(A,A,N) and ETS(A,Ad,N).
D15 convergence checked: Whether a run's script read any optimiser status (for statsforecast, the iteration count model_["fit"].nit) and counted fits that did not converge.
D16 output tokens: Model output tokens of an agent over its whole run, summed from the usage of every model call in its transcript.
D17 active time: Time an agent spent working in a run, excluding waits for answers, as reported by the harness.
D18 record anchored: Whether the agent anchored the finished record (claimset lab anchor).
D19 runs written as the human: Number of run events an agent recorded with --by the human's actor ID.
D33 decision thresholds: The thresholds a lab record's confirmatory plans fix in their decision rules, as the plans state them; none when a record has no plan.
D34 hypotheses about an estimate: Number of a lab record's hypotheses that state the direction or size of an estimate (a bias towards zero, an estimate on a bound, a bias of about −0.04) rather than a pass-or-fail threshold.
Study design 3
D7 lab phase: Phase 1: four runs, two of the Sonnet agent and two of the Haiku agent, each in its own git repository with a local remote, given the same task ("Do the study in brief.md with claimset-lab, following its skill, from the first question to the finished record. Ask the human anything you need through me."). The orchestrating agent relays questions and answers them only from the fixed answers, gives no other help, and never shows an agent another run or the answer key. Runs in parallel on one 4-core machine.
D9 · part of D7, D8 fixed answers: The answers the orchestrating agent gives in place of the human, the same to every run (study/human.md): name and ORCID iD; licence CC-BY-4.0; funding none; the competing interest of the brief; "Approved." for a plan that names its quantities and decision rule before the data are run, otherwise "Please state the quantities and the decision rule first."; for highlights "Choose up to three principal claims you consider most important."; for anything else "Use your judgement and say what you decided."
D8 claimset phase: Phase 2: the lab record of run lab-sonnet-1 (the only one that verifies, planned before its data and fixed statsforecast's model form) copied to four folders, with the draft its agent had exported removed; two Sonnet agent and two Haiku agent runs, each told "Export this lab record with claimset lab export and complete the claimset for submission, following the claimset-submit skill. Do not submit it." Questions answered as in the lab phase.
Settings 1
D6 ETS estimation brief: The research task given to every agent (study/brief.md): compare how well smooth 1.2.0 (ES) and statsforecast 2.1.1 (AutoETS with a fixed model form) estimate the smoothing parameters of the ETS model that generated simulated data, by bias, RMSE and MAE and by whether the RMSE falls as the series length grows; five processes (ETS(A,N,N) with α 0.1 and 0.5, ETS(A,A,N) with (α, β) (0.3, 0.05) and (0.5, 0.2), ETS(A,N,A) with m = 12 and (α, γ) (0.2, 0.1)), series lengths 50, 100, 250 and 500, 300 replicates per cell, smooth's default backcasting as a separate arm, failures counted.
Context
Artefacts
- A1 · data The eight runs as the agents left them: each folder's files without its git directory, and its full git history (study branch included) in repo.bundle. runs
- A2 · trace The eight agents' transcripts (Claude Code subagent JSONL), with every tool call and result and the usage of every model call. transcripts
- A3 · prompt The task prompts given to the agents in each phase. prompts.md
- A4 · protocol The study brief and the fixed answers given in place of the human. study
- A5 · data Setup notes, every question and answer, agent IDs and usage per turn. log
- A8 · data Scoring per run, rubric scores as JSON, summary.md and report.md. results
- A9 · code The scoring scripts transcript.py, lab_summary.py and validate_runs.py. tools
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.3): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C2: 78, Partly supported
Novelty: new, by the authors
- −10 Open problems Limitation L1 (could reverse, not mitigated): Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Assumption C22 is not checked: The agents under test, run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −3 Open problems Limitation L2 (could bias magnitude, not mitigated): The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.