The claim
Under EntryLib rubric 0.2, a claimset that states no limitations and gives its claims no scope (claimset-haiku-2) gets a higher reliability score than one written from the same lab record with scoped claims and three specific limitations (claimset-sonnet-1).
Scope: conditions: entrylib-rubric 0.2.0, without --novelty, the six claimsets of this study, claimsets completed from one lab record; not for: other rubric versions · Data: E2 (T1.r1, T1.r4), artefacts A8, A10 · Uses: C13 · Limitations: L5 · Addresses: B5
Claims it rests on
In EntryLib rubric 0.2, an unmitigated limitation that could bias a claim's magnitude costs the claim 12 points, more than a missing scope (10 points).
Scope: conditions: entrylib-rubric 0.2.0, without --novelty, the six claimsets of this study, costs per claim; not for: other rubric versions · Data: E4 (T1.r1, T1.r2), artefacts A10 · Used by: C11
Evidence
The claimset phase: two runs each of the Sonnet agent and the Haiku agent exported one lab record and completed the claimset with claimset-submit. For each final claimset the orchestrating agent ran claimset validate and claimset score (EntryLib rubric 0.2), counted the validator outcomes in the transcript (tools/validate_runs.py), checked that no record-derived field (hypothesis, planning, planned_at, test, exploration counts) had changed, and scored every claim against the answer key (table T1). Table T2 scores, under the same rubric, the two claimsets that lab-phase agents wrote on their own records after finishing them (beyond their task).
Setup: studied Sonnet agent, Haiku agent · cases ETS estimation brief · procedure claimset phase, fixed answers, claimset-submit · lens answer key, EntryLib rubric 0.2, claimset validates, validator outcomes until valid, warnings left, reliability score, claim correctness, answer-key findings stated, principal claims with scope, limitations, research agent's model kept, interventions, output tokens
| Row | run | agent | claimset validates | validator outcomes until valid | warnings left | reliability score | claim correctness (outcome: correct) | claim correctness (outcome: partly correct) | claim correctness (outcome: wrong) | answer-key findings stated | principal claims with scope | limitations | research agent's model kept | interventions | output tokens |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| r1 | claimset-sonnet-1 | Sonnet agent | yes | 4 | 2 | 73 | 4 | 1 | 0 | 0 | 3 | 3 | yes | 0 | 50493 |
| r2 | claimset-sonnet-2 | Sonnet agent | yes | 5 | 3 | 72 | 5 | 0 | 0 | 0 | 3 | 3 | yes | 0 | 44792 |
| r3 | claimset-haiku-1 | Haiku agent | yes | 6 | 1 | 79 | 4 | 1 | 0 | 0 | 0 | 3 | no | 0 | 69641 |
| r4 | claimset-haiku-2 | Haiku agent | yes | 4 | 11 | 78 | 3 | 2 | 0 | 0 | 0 | 0 | yes | 1 | 55371 |
| Row | run | agent | planning of the principal claims | reliability score | limitations | principal claims with scope |
|---|---|---|---|---|---|---|
| r1 | lab-sonnet-2 | Sonnet agent | confirmatory | 73 | 4 | 3 |
| r2 | lab-haiku-1 | Haiku agent | exploratory | 92 | 0 | 3 |
A reading of the rubric's code: the costs EntryLib rubric 0.2 deducts for the problems that decide the scores compared in this paper.
Setup: studied EntryLib rubric 0.2 · cases EntryLib rubric 0.2 · procedure reading of the rubric's code · lens rubric cost
| Row | problem | rubric cost |
|---|---|---|
| r1 | a limitation that could bias magnitude, not mitigated | 12 |
| r2 | no scope | 10 |
| r3 | a plan anchored only by a git push | 3 |
| r4 | a descriptive claim without a plan | 0 |
Limitations
One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322).
Definitions
Methods 5
D2 claimset-submit: The claimset-submit skill of claimset 0.41.6 (claimset specification v0.41), together with claimset lab export, claimset validate and claimset render: it turns a lab record into a claimset and completes it for submission.
D5 agents under test: The two agent configurations compared: the Sonnet agent and the Haiku agent. Members: D3, D4.
D3 · part of D5 Sonnet agent: A Claude Code 2.1.322 subagent started with the Agent tool (type general-purpose) on the model claude-sonnet-5-5, working in a folder of its own that holds the study brief, AGENTS.md and the claimset skills as files; it asks the human by ending its turn with its questions.
D4 · part of D5 Haiku agent: The same as the Sonnet agent, on the model claude-haiku-5-5.
D23 EntryLib rubric 0.2: The scoring rubric EntryLib applies to published claimsets (entrylib-rubric 0.2.0), run on this machine with claimset score and without --novelty: each principal claim starts at 100 and loses fixed costs for its problems (support, plan, checkability, context, open limitations and assumptions); the paper's score is that of its lowest principal claim.
Data 1
D10 answer key: The findings a correct paper on the brief would state, with their numbers (reference/answer_key.md), from a reference simulation of the full design written by the orchestrating agent before Phase 1 without claimset-lab (reference/simulate.py, optcheck.py, damped_check.py). Amended once during Phase 1 (statsforecast's β cap).
Variables and measures 12
D16 output tokens: Model output tokens of an agent over its whole run, summed from the usage of every model call in its transcript.
D20 claimset validates: Whether claimset validate reports the final claimset valid (no errors).
D21 validator outcomes until valid: The position, among the validator outcomes a claimset run saw (the export's own summary included), of the first with no errors.
D22 warnings left: Validator warnings on the final claimset.
D24 reliability score: The paper score of a claimset under EntryLib rubric 0.2, 0–100.
D25 claim correctness: Each claim of a claimset checked against the answer key: correct (direction and magnitude agree within Monte Carlo error), partly correct (direction agrees; magnitude, scope or a qualifier does not), wrong, or not supported by the record.
D26 answer-key findings stated: Number of the answer key's findings on bias (K2–K5), statsforecast's non-convergence (K1.2) and statsforecast's β cap (K8) that a claimset states as claims.
D27 principal claims with scope: Number of a claimset's principal claims that have a scope.
D28 limitations: Number of limitations a claimset records.
D29 research agent's model kept: Whether a completed claimset keeps the model of the agent that did the research, as the lab record names it.
D30 interventions: Number of messages the orchestrating agent sent a run that were not answers from the fixed answers.
D35 rubric cost: Points that EntryLib rubric 0.2 deducts from a claim's score for one problem, as its file rubric.json states them.
Study design 4
D7 lab phase: Phase 1: four runs, two of the Sonnet agent and two of the Haiku agent, each in its own git repository with a local remote, given the same task ("Do the study in brief.md with claimset-lab, following its skill, from the first question to the finished record. Ask the human anything you need through me."). The orchestrating agent relays questions and answers them only from the fixed answers, gives no other help, and never shows an agent another run or the answer key. Runs in parallel on one 4-core machine.
D9 · part of D7, D8 fixed answers: The answers the orchestrating agent gives in place of the human, the same to every run (study/human.md): name and ORCID iD; licence CC-BY-4.0; funding none; the competing interest of the brief; "Approved." for a plan that names its quantities and decision rule before the data are run, otherwise "Please state the quantities and the decision rule first."; for highlights "Choose up to three principal claims you consider most important."; for anything else "Use your judgement and say what you decided."
D8 claimset phase: Phase 2: the lab record of run lab-sonnet-1 (the only one that verifies, planned before its data and fixed statsforecast's model form) copied to four folders, with the draft its agent had exported removed; two Sonnet agent and two Haiku agent runs, each told "Export this lab record with claimset lab export and complete the claimset for submission, following the claimset-submit skill. Do not submit it." Questions answered as in the lab phase.
D36 reading of the rubric's code: The orchestrating agent read the costs in src/rubric/rubric.json and the rule that applies the plan check in src/rubric/score.py (lines 28 and 111), at the commit artefact A10 names.
Settings 1
D6 ETS estimation brief: The research task given to every agent (study/brief.md): compare how well smooth 1.2.0 (ES) and statsforecast 2.1.1 (AutoETS with a fixed model form) estimate the smoothing parameters of the ETS model that generated simulated data, by bias, RMSE and MAE and by whether the RMSE falls as the series length grows; five processes (ETS(A,N,N) with α 0.1 and 0.5, ETS(A,A,N) with (α, β) (0.3, 0.05) and (0.5, 0.2), ETS(A,N,A) with m = 12 and (α, γ) (0.2, 0.1)), series lengths 50, 100, 250 and 500, 300 replicates per cell, smooth's default backcasting as a separate arm, failures counted.
Context
Artefacts
- A1 · data The eight runs as the agents left them: each folder's files without its git directory, and its full git history (study branch included) in repo.bundle. runs
- A2 · trace The eight agents' transcripts (Claude Code subagent JSONL), with every tool call and result and the usage of every model call. transcripts
- A3 · prompt The task prompts given to the agents in each phase. prompts.md
- A6 · data The reference analysis: estimates.csv, summary.csv, consistency.csv, optcheck.csv, damped_check.csv and answer_key.md. reference
- A8 · data Scoring per run, rubric scores as JSON, summary.md and report.md. results
- A9 · code The scoring scripts transcript.py, lab_summary.py and validate_runs.py. tools
- A10 · software EntryLib's scoring rubric, entrylib-rubric 0.2.0 (src/rubric/rubric.json holds the costs; src/rubric/score.py applies them). 8a6f9f948a83c0a6f8c60c1e101ed5c4f5399caf
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.3): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C11: 94, Well supported
Novelty: new, by the authors
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C13: 100, Well supported
Nothing lost.