The claim
None of the four claimsets that the agents under test completed with claimset-submit states as a claim any of the main findings of the answer key: how large the estimation bias is, that statsforecast's optimiser fails to converge for the seasonal model, or that statsforecast caps β (answer-key findings stated).
Scope: data: agent runs on one simulation study task; conditions: one lab record (lab-sonnet-1) completed four times, claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, claims only, not limitations or open questions; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E2 (T1), artefacts A6, A8 · Uses: C12 · Assumes: C21 · Limitations: L3, L4, L5 · Addresses: B6
Claims it rests on
The lab record that the claimset phase started from holds three hypotheses, each a threshold test on the RMSE (a band for the ratio of the two packages, backcasting better in most cells at the shortest length, the longest length against the shortest), and no hypothesis about the size of the bias or the RMSE.
Scope: conditions: the lab record of run lab-sonnet-1, 1,130 events · Data: E2, artefacts A1 · Used by: C7
Evidence
The claimset phase: two runs each of the Sonnet agent and the Haiku agent exported one lab record and completed the claimset with claimset-submit. For each final claimset the orchestrating agent ran claimset validate and claimset score (EntryLib rubric 0.2), counted the validator outcomes in the transcript (tools/validate_runs.py), checked that no record-derived field (hypothesis, planning, planned_at, test, exploration counts) had changed, and scored every claim against the answer key (table T1). Table T2 scores, under the same rubric, the two claimsets that lab-phase agents wrote on their own records after finishing them (beyond their task).
Setup: studied Sonnet agent, Haiku agent · cases ETS estimation brief · procedure claimset phase, fixed answers, claimset-submit · lens answer key, EntryLib rubric 0.2, claimset validates, validator outcomes until valid, warnings left, reliability score, claim correctness, answer-key findings stated, principal claims with scope, limitations, research agent's model kept, interventions, output tokens
| Row | run | agent | claimset validates | validator outcomes until valid | warnings left | reliability score | claim correctness (outcome: correct) | claim correctness (outcome: partly correct) | claim correctness (outcome: wrong) | answer-key findings stated | principal claims with scope | limitations | research agent's model kept | interventions | output tokens |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| r1 | claimset-sonnet-1 | Sonnet agent | yes | 4 | 2 | 73 | 4 | 1 | 0 | 0 | 3 | 3 | yes | 0 | 50493 |
| r2 | claimset-sonnet-2 | Sonnet agent | yes | 5 | 3 | 72 | 5 | 0 | 0 | 0 | 3 | 3 | yes | 0 | 44792 |
| r3 | claimset-haiku-1 | Haiku agent | yes | 6 | 1 | 79 | 4 | 1 | 0 | 0 | 0 | 3 | no | 0 | 69641 |
| r4 | claimset-haiku-2 | Haiku agent | yes | 4 | 11 | 78 | 3 | 2 | 0 | 0 | 0 | 0 | yes | 1 | 55371 |
| Row | run | agent | planning of the principal claims | reliability score | limitations | principal claims with scope |
|---|---|---|---|---|---|---|
| r1 | lab-sonnet-2 | Sonnet agent | confirmatory | 73 | 4 | 3 |
| r2 | lab-haiku-1 | Haiku agent | exploratory | 92 | 0 | 3 |
The reference simulation of the brief (R = 300 per cell, 20 cells, three arms, 110 s on 4 cores), with optcheck.py and damped_check.py. Table T1: statsforecast's β estimates for ETS(A,A,N) with (α, β) = (0.5, 0.2), against smooth's; table T2: statsforecast fits of ETS(A,N,A) that stopped at the optimiser's iteration cap. The β cap was first reported by run lab-haiku-1 (its exploratory run RN1); it was verified here against the statsforecast 2.1.1 source and these estimates, and added to the answer key.
Setup: studied reference simulation · cases ETS estimation brief · procedure reference simulation · lens statsforecast β cap, statsforecast iteration cap
| Row | series length | largest statsforecast β estimate | statsforecast fits within 0.001 of 0.20006 | smooth fits with β above 0.20006 |
|---|---|---|---|---|
| r1 | 50 | 0.2001 | 100 | 131 |
| r2 | 100 | 0.2001 | 135 | 146 |
| r3 | 250 | 0.2001 | 132 | 132 |
| r4 | 500 | 0.2001 | 144 | 137 |
| Row | series length | statsforecast iteration cap |
|---|---|---|
| r1 | 50 | 288 |
| r2 | 100 | 296 |
| r3 | 250 | 300 |
| r4 | 500 | 299 |
Limitations
One scorer, the orchestrating agent, wrote the answer key and the protocol and scored every claim, knowing which agent wrote it.
The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key.
One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322).
Assumptions
The answer key states the findings that a correct paper on the brief would state, with correct numbers. Rested on by C7.
Definitions
Methods 5
D2 claimset-submit: The claimset-submit skill of claimset 0.41.6 (claimset specification v0.41), together with claimset lab export, claimset validate and claimset render: it turns a lab record into a claimset and completes it for submission.
D5 agents under test: The two agent configurations compared: the Sonnet agent and the Haiku agent. Members: D3, D4.
D3 · part of D5 Sonnet agent: A Claude Code 2.1.322 subagent started with the Agent tool (type general-purpose) on the model claude-sonnet-5-5, working in a folder of its own that holds the study brief, AGENTS.md and the claimset skills as files; it asks the human by ending its turn with its questions.
D4 · part of D5 Haiku agent: The same as the Sonnet agent, on the model claude-haiku-5-5.
D23 EntryLib rubric 0.2: The scoring rubric EntryLib applies to published claimsets (entrylib-rubric 0.2.0), run on this machine with claimset score and without --novelty: each principal claim starts at 100 and loses fixed costs for its problems (support, plan, checkability, context, open limitations and assumptions); the paper's score is that of its lowest principal claim.
Data 1
D10 answer key: The findings a correct paper on the brief would state, with their numbers (reference/answer_key.md), from a reference simulation of the full design written by the orchestrating agent before Phase 1 without claimset-lab (reference/simulate.py, optcheck.py, damped_check.py). Amended once during Phase 1 (statsforecast's β cap).
Variables and measures 12
D16 output tokens: Model output tokens of an agent over its whole run, summed from the usage of every model call in its transcript.
D20 claimset validates: Whether claimset validate reports the final claimset valid (no errors).
D21 validator outcomes until valid: The position, among the validator outcomes a claimset run saw (the export's own summary included), of the first with no errors.
D22 warnings left: Validator warnings on the final claimset.
D24 reliability score: The paper score of a claimset under EntryLib rubric 0.2, 0–100.
D25 claim correctness: Each claim of a claimset checked against the answer key: correct (direction and magnitude agree within Monte Carlo error), partly correct (direction agrees; magnitude, scope or a qualifier does not), wrong, or not supported by the record.
D26 answer-key findings stated: Number of the answer key's findings on bias (K2–K5), statsforecast's non-convergence (K1.2) and statsforecast's β cap (K8) that a claimset states as claims.
D27 principal claims with scope: Number of a claimset's principal claims that have a scope.
D28 limitations: Number of limitations a claimset records.
D29 research agent's model kept: Whether a completed claimset keeps the model of the agent that did the research, as the lab record names it.
D30 interventions: Number of messages the orchestrating agent sent a run that were not answers from the fixed answers.
D37 statsforecast iteration cap: Number of statsforecast fits whose optimiser stopped at its iteration cap (nit = 1001) in the reference simulation.
Study design 4
D7 lab phase: Phase 1: four runs, two of the Sonnet agent and two of the Haiku agent, each in its own git repository with a local remote, given the same task ("Do the study in brief.md with claimset-lab, following its skill, from the first question to the finished record. Ask the human anything you need through me."). The orchestrating agent relays questions and answers them only from the fixed answers, gives no other help, and never shows an agent another run or the answer key. Runs in parallel on one 4-core machine.
D9 · part of D7, D8 fixed answers: The answers the orchestrating agent gives in place of the human, the same to every run (study/human.md): name and ORCID iD; licence CC-BY-4.0; funding none; the competing interest of the brief; "Approved." for a plan that names its quantities and decision rule before the data are run, otherwise "Please state the quantities and the decision rule first."; for highlights "Choose up to three principal claims you consider most important."; for anything else "Use your judgement and say what you decided."
D8 claimset phase: Phase 2: the lab record of run lab-sonnet-1 (the only one that verifies, planned before its data and fixed statsforecast's model form) copied to four folders, with the draft its agent had exported removed; two Sonnet agent and two Haiku agent runs, each told "Export this lab record with claimset lab export and complete the claimset for submission, following the claimset-submit skill. Do not submit it." Questions answered as in the lab phase.
D31 reference simulation: The reference analysis behind the answer key: 300 series per cell of the brief, each fitted by smooth with optimised initial states, smooth with backcasting and statsforecast AutoETS(model, season_length, damped=False) with its defaults (likelihood, bounds both), seeds numpy.random.default_rng([20261111, process, T, replicate]).
Settings 1
D6 ETS estimation brief: The research task given to every agent (study/brief.md): compare how well smooth 1.2.0 (ES) and statsforecast 2.1.1 (AutoETS with a fixed model form) estimate the smoothing parameters of the ETS model that generated simulated data, by bias, RMSE and MAE and by whether the RMSE falls as the series length grows; five processes (ETS(A,N,N) with α 0.1 and 0.5, ETS(A,A,N) with (α, β) (0.3, 0.05) and (0.5, 0.2), ETS(A,N,A) with m = 12 and (α, γ) (0.2, 0.1)), series lengths 50, 100, 250 and 500, 300 replicates per cell, smooth's default backcasting as a separate arm, failures counted.
Concepts 1
D32 statsforecast β cap: In statsforecast 2.1.1 (statsforecast.ets.initparam), when α is estimated its starting value is (0.20006 for ), and the upper bound of β is set to for the whole optimisation, so β cannot exceed .
Context
Artefacts
- A1 · data The eight runs as the agents left them: each folder's files without its git directory, and its full git history (study branch included) in repo.bundle. runs
- A2 · trace The eight agents' transcripts (Claude Code subagent JSONL), with every tool call and result and the usage of every model call. transcripts
- A3 · prompt The task prompts given to the agents in each phase. prompts.md
- A6 · data The reference analysis: estimates.csv, summary.csv, consistency.csv, optcheck.csv, damped_check.csv and answer_key.md. reference
- A7 · code The reference analysis scripts simulate.py, optcheck.py and damped_check.py. reference
- A8 · data Scoring per run, rubric scores as JSON, summary.md and report.md. results
- A9 · code The scoring scripts transcript.py, lab_summary.py and validate_runs.py. tools
- A10 · software EntryLib's scoring rubric, entrylib-rubric 0.2.0 (src/rubric/rubric.json holds the costs; src/rubric/score.py applies them). 8a6f9f948a83c0a6f8c60c1e101ed5c4f5399caf
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.3): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C7: 74, Partly supported
Novelty: new, by the authors
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −6 Open problems Limitation L4 (could bias magnitude, not mitigated): The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Assumption C21 is not checked: The answer key states the findings that a correct paper on the brief would state, with correct numbers. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Limitation L3 (could bias magnitude, mitigation partial): One scorer, the orchestrating agent, wrote the answer key and the protocol and scored every claim, knowing which agent wrote it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C12: 90, Well supported
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.