Claude Sonnet and Claude Haiku agents keeping Claimset lab records and completing claimsets: a pilot on one simulation study · C8

D1 claimset-lab: The claimset-lab skill (.claude/skills/claimset-lab/SKILL.md, installed by claimset init --agent claude) and the claimset lab commands of claimset 0.41.6, which keep a lab record (lab record specification v0.21): an append-only, hash-chained log of questions, hypotheses, plans, approvals, runs, results and verdicts, kept on a study branch and pushed after every command.

D2 claimset-submit: The claimset-submit skill of claimset 0.41.6 (claimset specification v0.41), together with claimset lab export, claimset validate and claimset render: it turns a lab record into a claimset and completes it for submission.

D5 agents under test: The two agent configurations compared: the Sonnet agent and the Haiku agent. Members: D3, D4.

D23 EntryLib rubric 0.2: The scoring rubric EntryLib applies to published claimsets (entrylib-rubric 0.2.0), run on this machine with claimset score and without --novelty: each principal claim starts at 100 and loses fixed costs for its problems (support, plan, checkability, context, open limitations and assumptions); the paper's score is that of its lowest principal claim.

D10 answer key: The findings a correct paper on the brief would state, with their numbers (reference/answer_key.md), from a reference simulation of the full design written by the orchestrating agent before Phase 1 without claimset-lab (reference/simulate.py, optcheck.py, damped_check.py). Amended once during Phase 1 (statsforecast's β cap).

D11 record verifies: Whether claimset lab verify reports the lab record OK at the end of a run (hash chain, rules, run files).

D12 tool refusals: Number of commands that claimset refused in a run with a lab rule (L1–L30) or an argument or overwrite error; validator errors on a claimset are counted separately.

D13 plans before data: Whether each confirmatory run of a lab record ran under a plan with its decision rule recorded, and approved by the human, before the run, and whether its hypotheses were written before any run informed them: all, all, hypotheses after pilots or none (no plan).

D14 model form fixed: Whether a run's simulation script fixed statsforecast's model form as the brief asks (damped=False); without it, AutoETS(model="AAN") and ets_f(..., model="AAN") choose between ETS(A,A,N) and ETS(A,Ad,N).

D15 convergence checked: Whether a run's script read any optimiser status (for statsforecast, the iteration count model_["fit"].nit) and counted fits that did not converge.

D16 output tokens: Model output tokens of an agent over its whole run, summed from the usage of every model call in its transcript.

D17 active time: Time an agent spent working in a run, excluding waits for answers, as reported by the harness.

D18 record anchored: Whether the agent anchored the finished record (claimset lab anchor).

D19 runs written as the human: Number of run events an agent recorded with --by the human's actor ID.

D20 claimset validates: Whether claimset validate reports the final claimset valid (no errors).

D21 validator outcomes until valid: The position, among the validator outcomes a claimset run saw (the export's own summary included), of the first with no errors.

D22 warnings left: Validator warnings on the final claimset.

D24 reliability score: The paper score of a claimset under EntryLib rubric 0.2, 0–100.

D25 claim correctness: Each claim of a claimset checked against the answer key: correct (direction and magnitude agree within Monte Carlo error), partly correct (direction agrees; magnitude, scope or a qualifier does not), wrong, or not supported by the record.

D26 answer-key findings stated: Number of the answer key's findings on bias (K2–K5), statsforecast's non-convergence (K1.2) and statsforecast's β cap (K8) that a claimset states as claims.

D27 principal claims with scope: Number of a claimset's principal claims that have a scope.

D28 limitations: Number of limitations a claimset records.

D29 research agent's model kept: Whether a completed claimset keeps the model of the agent that did the research, as the lab record names it.

D30 interventions: Number of messages the orchestrating agent sent a run that were not answers from the fixed answers.

D33 decision thresholds: The thresholds a lab record's confirmatory plans fix in their decision rules, as the plans state them; none when a record has no plan.

D34 hypotheses about an estimate: Number of a lab record's hypotheses that state the direction or size of an estimate (a bias towards zero, an estimate on a bound, a bias of about −0.04) rather than a pass-or-fail threshold.

D37 statsforecast iteration cap: Number of statsforecast fits whose optimiser stopped at its iteration cap (nit = 1001) in the reference simulation.

D7 lab phase: Phase 1: four runs, two of the Sonnet agent and two of the Haiku agent, each in its own git repository with a local remote, given the same task ("Do the study in brief.md with claimset-lab, following its skill, from the first question to the finished record. Ask the human anything you need through me."). The orchestrating agent relays questions and answers them only from the fixed answers, gives no other help, and never shows an agent another run or the answer key. Runs in parallel on one 4-core machine.

D8 claimset phase: Phase 2: the lab record of run lab-sonnet-1 (the only one that verifies, planned before its data and fixed statsforecast's model form) copied to four folders, with the draft its agent had exported removed; two Sonnet agent and two Haiku agent runs, each told "Export this lab record with claimset lab export and complete the claimset for submission, following the claimset-submit skill. Do not submit it." Questions answered as in the lab phase.

D31 reference simulation: The reference analysis behind the answer key: 300 series per cell of the brief, each fitted by smooth with optimised initial states, smooth with backcasting and statsforecast AutoETS(model, season_length, damped=False) with its defaults (likelihood, bounds both), seeds numpy.random.default_rng([20261111, process, T, replicate]).

D6 ETS estimation brief: The research task given to every agent (study/brief.md): compare how well smooth 1.2.0 (ES) and statsforecast 2.1.1 (AutoETS with a fixed model form) estimate the smoothing parameters of the ETS model that generated simulated data, by bias, RMSE and MAE and by whether the RMSE falls as the series length grows; five processes (ETS(A,N,N) with α 0.1 and 0.5, ETS(A,A,N) with (α, β) (0.3, 0.05) and (0.5, 0.2), ETS(A,N,A) with m = 12 and (α, γ) (0.2, 0.1)), series lengths 50, 100, 250 and 500, 300 replicates per cell, smooth's default backcasting as a separate arm, failures counted.

D32 statsforecast β cap: In statsforecast 2.1.1 (statsforecast.ets.initparam), when α is estimated its starting value is $$\alpha_0 = 0.0001 + 0.2 \cdot \frac{0.9999 - 0.0001}{m}$$ (0.20006 for $m = 1$), and the upper bound of β is set to $\min(0.9999, \alpha_0)$ for the whole optimisation, so β cannot exceed $\alpha_0$.