The claim
Every claimset that the agents under test completed with claimset-submit from one lab record validates with no errors (claimset validates), at the fourth to sixth validator outcome (validator outcomes until valid).
Scope: data: agent runs on one simulation study task; conditions: one lab record (lab-sonnet-1) completed four times, claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the final file of each claimset run; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E2 (T1), artefacts A1, A2 · Assumes: C22 · Limitations: L2, L5, L6 · Addresses: B6
Evidence
The claimset phase: two runs each of the Sonnet agent and the Haiku agent exported one lab record and completed the claimset with claimset-submit. For each final claimset the orchestrating agent ran claimset validate and claimset score (EntryLib rubric 0.2), counted the validator outcomes in the transcript (tools/validate_runs.py), checked that no record-derived field (hypothesis, planning, planned_at, test, exploration counts) had changed, and scored every claim against the answer key (table T1). Table T2 scores, under the same rubric, the two claimsets that lab-phase agents wrote on their own records after finishing them (beyond their task).
Setup: studied Sonnet agent, Haiku agent · cases ETS estimation brief · procedure claimset phase, fixed answers, claimset-submit · lens answer key, EntryLib rubric 0.2, claimset validates, validator outcomes until valid, warnings left, reliability score, claim correctness, answer-key findings stated, principal claims with scope, limitations, research agent's model kept, interventions, output tokens
| Row | run | agent | claimset validates | validator outcomes until valid | warnings left | reliability score | claim correctness (outcome: correct) | claim correctness (outcome: partly correct) | claim correctness (outcome: wrong) | answer-key findings stated | principal claims with scope | limitations | research agent's model kept | interventions | output tokens |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| r1 | claimset-sonnet-1 | Sonnet agent | yes | 4 | 2 | 73 | 4 | 1 | 0 | 0 | 3 | 3 | yes | 0 | 50493 |
| r2 | claimset-sonnet-2 | Sonnet agent | yes | 5 | 3 | 72 | 5 | 0 | 0 | 0 | 3 | 3 | yes | 0 | 44792 |
| r3 | claimset-haiku-1 | Haiku agent | yes | 6 | 1 | 79 | 4 | 1 | 0 | 0 | 0 | 3 | no | 0 | 69641 |
| r4 | claimset-haiku-2 | Haiku agent | yes | 4 | 11 | 78 | 3 | 2 | 0 | 0 | 0 | 0 | yes | 1 | 55371 |
| Row | run | agent | planning of the principal claims | reliability score | limitations | principal claims with scope |
|---|---|---|---|---|---|---|
| r1 | lab-sonnet-2 | Sonnet agent | confirmatory | 73 | 4 | 3 |
| r2 | lab-haiku-1 | Haiku agent | exploratory | 92 | 0 | 3 |
Limitations
The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine.
One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322).
One claimset run (claimset-haiku-2) stopped with its own stated work undone and was sent one message with no content ("Continue.") before it finished.
Assumptions
The agents under test, run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders. Rested on by C5.
Definitions
Methods 5
D2 claimset-submit: The claimset-submit skill of claimset 0.41.6 (claimset specification v0.41), together with claimset lab export, claimset validate and claimset render: it turns a lab record into a claimset and completes it for submission.
D5 agents under test: The two agent configurations compared: the Sonnet agent and the Haiku agent. Members: D3, D4.
D3 · part of D5 Sonnet agent: A Claude Code 2.1.322 subagent started with the Agent tool (type general-purpose) on the model claude-sonnet-5-5, working in a folder of its own that holds the study brief, AGENTS.md and the claimset skills as files; it asks the human by ending its turn with its questions.
D4 · part of D5 Haiku agent: The same as the Sonnet agent, on the model claude-haiku-5-5.
D23 EntryLib rubric 0.2: The scoring rubric EntryLib applies to published claimsets (entrylib-rubric 0.2.0), run on this machine with claimset score and without --novelty: each principal claim starts at 100 and loses fixed costs for its problems (support, plan, checkability, context, open limitations and assumptions); the paper's score is that of its lowest principal claim.
Data 1
D10 answer key: The findings a correct paper on the brief would state, with their numbers (reference/answer_key.md), from a reference simulation of the full design written by the orchestrating agent before Phase 1 without claimset-lab (reference/simulate.py, optcheck.py, damped_check.py). Amended once during Phase 1 (statsforecast's β cap).
Variables and measures 11
D16 output tokens: Model output tokens of an agent over its whole run, summed from the usage of every model call in its transcript.
D20 claimset validates: Whether claimset validate reports the final claimset valid (no errors).
D21 validator outcomes until valid: The position, among the validator outcomes a claimset run saw (the export's own summary included), of the first with no errors.
D22 warnings left: Validator warnings on the final claimset.
D24 reliability score: The paper score of a claimset under EntryLib rubric 0.2, 0–100.
D25 claim correctness: Each claim of a claimset checked against the answer key: correct (direction and magnitude agree within Monte Carlo error), partly correct (direction agrees; magnitude, scope or a qualifier does not), wrong, or not supported by the record.
D26 answer-key findings stated: Number of the answer key's findings on bias (K2–K5), statsforecast's non-convergence (K1.2) and statsforecast's β cap (K8) that a claimset states as claims.
D27 principal claims with scope: Number of a claimset's principal claims that have a scope.
D28 limitations: Number of limitations a claimset records.
D29 research agent's model kept: Whether a completed claimset keeps the model of the agent that did the research, as the lab record names it.
D30 interventions: Number of messages the orchestrating agent sent a run that were not answers from the fixed answers.
Study design 3
D7 lab phase: Phase 1: four runs, two of the Sonnet agent and two of the Haiku agent, each in its own git repository with a local remote, given the same task ("Do the study in brief.md with claimset-lab, following its skill, from the first question to the finished record. Ask the human anything you need through me."). The orchestrating agent relays questions and answers them only from the fixed answers, gives no other help, and never shows an agent another run or the answer key. Runs in parallel on one 4-core machine.
D9 · part of D7, D8 fixed answers: The answers the orchestrating agent gives in place of the human, the same to every run (study/human.md): name and ORCID iD; licence CC-BY-4.0; funding none; the competing interest of the brief; "Approved." for a plan that names its quantities and decision rule before the data are run, otherwise "Please state the quantities and the decision rule first."; for highlights "Choose up to three principal claims you consider most important."; for anything else "Use your judgement and say what you decided."
D8 claimset phase: Phase 2: the lab record of run lab-sonnet-1 (the only one that verifies, planned before its data and fixed statsforecast's model form) copied to four folders, with the draft its agent had exported removed; two Sonnet agent and two Haiku agent runs, each told "Export this lab record with claimset lab export and complete the claimset for submission, following the claimset-submit skill. Do not submit it." Questions answered as in the lab phase.
Settings 1
D6 ETS estimation brief: The research task given to every agent (study/brief.md): compare how well smooth 1.2.0 (ES) and statsforecast 2.1.1 (AutoETS with a fixed model form) estimate the smoothing parameters of the ETS model that generated simulated data, by bias, RMSE and MAE and by whether the RMSE falls as the series length grows; five processes (ETS(A,N,N) with α 0.1 and 0.5, ETS(A,A,N) with (α, β) (0.3, 0.05) and (0.5, 0.2), ETS(A,N,A) with m = 12 and (α, γ) (0.2, 0.1)), series lengths 50, 100, 250 and 500, 300 replicates per cell, smooth's default backcasting as a separate arm, failures counted.
Context
Artefacts
- A1 · data The eight runs as the agents left them: each folder's files without its git directory, and its full git history (study branch included) in repo.bundle. runs
- A2 · trace The eight agents' transcripts (Claude Code subagent JSONL), with every tool call and result and the usage of every model call. transcripts
- A3 · prompt The task prompts given to the agents in each phase. prompts.md
- A6 · data The reference analysis: estimates.csv, summary.csv, consistency.csv, optcheck.csv, damped_check.csv and answer_key.md. reference
- A8 · data Scoring per run, rubric scores as JSON, summary.md and report.md. results
- A9 · code The scoring scripts transcript.py, lab_summary.py and validate_runs.py. tools
- A10 · software EntryLib's scoring rubric, entrylib-rubric 0.2.0 (src/rubric/rubric.json holds the costs; src/rubric/score.py applies them). 8a6f9f948a83c0a6f8c60c1e101ed5c4f5399caf
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.3): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C5: 72, Partly supported
Novelty: new, by the authors
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −6 Open problems Limitation L2 (could bias magnitude, not mitigated): The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Limitation L6 (could bias magnitude, not mitigated): One claimset run (claimset-haiku-2) stopped with its own stated work undone and was sent one message with no content ("Continue.") before it finished. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −3 Open problems Assumption C22 is not checked: The agents under test, run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.