Summary
Can smaller AI models run a research project with a Claimset lab record and write it up as a claimset? Two Claude Sonnet and two Claude Haiku agents each did one simulation study, comparing how well two Python packages estimate exponential smoothing parameters; four more agents completed a claimset from one of the resulting records. All four lab records passed the tool's own verification and all four claimsets were valid. The Sonnet agents planned their tests before seeing data; the Haiku agents did not. No claim was wrong, but no claimset stated the study's main findings: the lab format turns only pass-or-fail hypotheses into claims, so the agents tested thresholds they chose themselves instead of reporting estimates. EntryLib's scoring rubric ranked a claimset without planned tests, and one without limitations or scope, above more careful ones. These are exploratory results from two runs per model on one task.
Highlights
- Support 67The agents under test answered the brief's questions about how large the bias and RMSE are with pass-or-fail tests against thresholds they chose themselves, so the estimates never became claims: claimset-lab turns only hypotheses decided by a rule into claims. C8
- Support 84Under EntryLib rubric 0.2, the one claimset that tested nothing under a plan gets a higher reliability score than each of the six claimsets whose principal claims were planned tests. C10
- Support 94Under EntryLib rubric 0.2, a claimset that states no limitations and gives its claims no scope (claimset-haiku-2) gets a higher reliability score than one written from the same lab record with scoped claims and three specific limitations (claimset-sonnet-1). C11
- Support 78Neither lab run of the Haiku agent planned its tests before seeing the data (plans before data): one run made no plan at all, and the other set its hypotheses and thresholds after three pilot runs. C3
- Support 74None of the four claimsets that the agents under test completed with claimset-submit states as a claim any of the main findings of the answer key: how large the estimation bias is, that statsforecast's optimiser fails to converge for the seasonal model, or that statsforecast caps β (answer-key findings stated). C7
Context
- Addresses 2408.06292 · from C1
- Agents that run research end to end, judged by an automated reviewer; here agents keep a verifiable lab record of one study, checked by the record's own verification.
- Addresses Nosek et al. (2018) · from C2
- Argues that predictions must be fixed before the data are seen; here the question is whether AI agents do so when a tool asks them to.
- Addresses Nosek et al. (2018) · from C3
- Argues that predictions must be fixed before the data are seen; here AI agents with the same tool did not, in either of two runs.
- Addresses Kaplan and Irvin (2015) · from C3
- Mandatory registration of trials was followed by more null results; here an agent that planned nothing reported more descriptive findings, on one task.
- Addresses 2408.06292 · from C5
- Agents write papers judged by an automated reviewer; here agents complete a claimset judged by a validator and against an answer key.
- Addresses 2408.06292 · from C6
- Agent-written papers judged by an automated reviewer; here each claim is checked against an answer key from a reference simulation.
- Addresses 2408.06292 · from C7
- Agent-written papers judged by an automated reviewer; here the check is which findings of a reference analysis the claims state.
- Addresses Cumming (2014) · from C8
- Recommends framing research questions as estimation rather than significance tests; here a tool's pass-or-fail plans led agents away from estimation.
- Addresses Scheel et al. (2021) · from C8
- Registered Reports report far fewer positive results than standard articles; here the registered format changed what agents reported, from estimates to threshold tests.
- Addresses Smaldino and McElreath (2016) · from C10
- Incentives that reward output select for weaker methods; here a scoring rubric ranks a claimset without planned tests above planned ones.
- Addresses Smaldino and McElreath (2016) · from C11
- Incentives that reward output select for weaker methods; here a scoring rubric ranks a claimset without limitations or scope above one with both.
Definitions
Methods 6
D1 claimset-lab: The claimset-lab skill (.claude/skills/claimset-lab/SKILL.md, installed by claimset init --agent claude) and the claimset lab commands of claimset 0.41.6, which keep a lab record (lab record specification v0.21): an append-only, hash-chained log of questions, hypotheses, plans, approvals, runs, results and verdicts, kept on a study branch and pushed after every command.
D2 claimset-submit: The claimset-submit skill of claimset 0.41.6 (claimset specification v0.41), together with claimset lab export, claimset validate and claimset render: it turns a lab record into a claimset and completes it for submission.
D5 agents under test: The two agent configurations compared: the Sonnet agent and the Haiku agent. Members: D3, D4.
D3 · part of D5 Sonnet agent: A Claude Code 2.1.322 subagent started with the Agent tool (type general-purpose) on the model claude-sonnet-5-5, working in a folder of its own that holds the study brief, AGENTS.md and the claimset skills as files; it asks the human by ending its turn with its questions.
D4 · part of D5 Haiku agent: The same as the Sonnet agent, on the model claude-haiku-5-5.
D23 EntryLib rubric 0.2: The scoring rubric EntryLib applies to published claimsets (entrylib-rubric 0.2.0), run on this machine with claimset score and without --novelty: each principal claim starts at 100 and loses fixed costs for its problems (support, plan, checkability, context, open limitations and assumptions); the paper's score is that of its lowest principal claim.
Data 1
D10 answer key: The findings a correct paper on the brief would state, with their numbers (reference/answer_key.md), from a reference simulation of the full design written by the orchestrating agent before Phase 1 without claimset-lab (reference/simulate.py, optcheck.py, damped_check.py). Amended once during Phase 1 (statsforecast's β cap).
Variables and measures 23
D11 record verifies: Whether claimset lab verify reports the lab record OK at the end of a run (hash chain, rules, run files).
D12 tool refusals: Number of commands that claimset refused in a run with a lab rule (L1–L30) or an argument or overwrite error; validator errors on a claimset are counted separately.
D13 plans before data: Whether each confirmatory run of a lab record ran under a plan with its decision rule recorded, and approved by the human, before the run, and whether its hypotheses were written before any run informed them: all, all, hypotheses after pilots or none (no plan).
D14 model form fixed: Whether a run's simulation script fixed statsforecast's model form as the brief asks (damped=False); without it, AutoETS(model="AAN") and ets_f(..., model="AAN") choose between ETS(A,A,N) and ETS(A,Ad,N).
D15 convergence checked: Whether a run's script read any optimiser status (for statsforecast, the iteration count model_["fit"].nit) and counted fits that did not converge.
D16 output tokens: Model output tokens of an agent over its whole run, summed from the usage of every model call in its transcript.
D17 active time: Time an agent spent working in a run, excluding waits for answers, as reported by the harness.
D18 record anchored: Whether the agent anchored the finished record (claimset lab anchor).
D19 runs written as the human: Number of run events an agent recorded with --by the human's actor ID.
D20 claimset validates: Whether claimset validate reports the final claimset valid (no errors).
D21 validator outcomes until valid: The position, among the validator outcomes a claimset run saw (the export's own summary included), of the first with no errors.
D22 warnings left: Validator warnings on the final claimset.
D24 reliability score: The paper score of a claimset under EntryLib rubric 0.2, 0–100.
D25 claim correctness: Each claim of a claimset checked against the answer key: correct (direction and magnitude agree within Monte Carlo error), partly correct (direction agrees; magnitude, scope or a qualifier does not), wrong, or not supported by the record.
D26 answer-key findings stated: Number of the answer key's findings on bias (K2–K5), statsforecast's non-convergence (K1.2) and statsforecast's β cap (K8) that a claimset states as claims.
D27 principal claims with scope: Number of a claimset's principal claims that have a scope.
D28 limitations: Number of limitations a claimset records.
D29 research agent's model kept: Whether a completed claimset keeps the model of the agent that did the research, as the lab record names it.
D30 interventions: Number of messages the orchestrating agent sent a run that were not answers from the fixed answers.
D33 decision thresholds: The thresholds a lab record's confirmatory plans fix in their decision rules, as the plans state them; none when a record has no plan.
D34 hypotheses about an estimate: Number of a lab record's hypotheses that state the direction or size of an estimate (a bias towards zero, an estimate on a bound, a bias of about −0.04) rather than a pass-or-fail threshold.
D35 rubric cost: Points that EntryLib rubric 0.2 deducts from a claim's score for one problem, as its file rubric.json states them.
D37 statsforecast iteration cap: Number of statsforecast fits whose optimiser stopped at its iteration cap (nit = 1001) in the reference simulation.
Study design 5
D7 lab phase: Phase 1: four runs, two of the Sonnet agent and two of the Haiku agent, each in its own git repository with a local remote, given the same task ("Do the study in brief.md with claimset-lab, following its skill, from the first question to the finished record. Ask the human anything you need through me."). The orchestrating agent relays questions and answers them only from the fixed answers, gives no other help, and never shows an agent another run or the answer key. Runs in parallel on one 4-core machine.
D9 · part of D7, D8 fixed answers: The answers the orchestrating agent gives in place of the human, the same to every run (study/human.md): name and ORCID iD; licence CC-BY-4.0; funding none; the competing interest of the brief; "Approved." for a plan that names its quantities and decision rule before the data are run, otherwise "Please state the quantities and the decision rule first."; for highlights "Choose up to three principal claims you consider most important."; for anything else "Use your judgement and say what you decided."
D8 claimset phase: Phase 2: the lab record of run lab-sonnet-1 (the only one that verifies, planned before its data and fixed statsforecast's model form) copied to four folders, with the draft its agent had exported removed; two Sonnet agent and two Haiku agent runs, each told "Export this lab record with claimset lab export and complete the claimset for submission, following the claimset-submit skill. Do not submit it." Questions answered as in the lab phase.
D31 reference simulation: The reference analysis behind the answer key: 300 series per cell of the brief, each fitted by smooth with optimised initial states, smooth with backcasting and statsforecast AutoETS(model, season_length, damped=False) with its defaults (likelihood, bounds both), seeds numpy.random.default_rng([20261111, process, T, replicate]).
D36 reading of the rubric's code: The orchestrating agent read the costs in src/rubric/rubric.json and the rule that applies the plan check in src/rubric/score.py (lines 28 and 111), at the commit artefact A10 names.
Settings 1
D6 ETS estimation brief: The research task given to every agent (study/brief.md): compare how well smooth 1.2.0 (ES) and statsforecast 2.1.1 (AutoETS with a fixed model form) estimate the smoothing parameters of the ETS model that generated simulated data, by bias, RMSE and MAE and by whether the RMSE falls as the series length grows; five processes (ETS(A,N,N) with α 0.1 and 0.5, ETS(A,A,N) with (α, β) (0.3, 0.05) and (0.5, 0.2), ETS(A,N,A) with m = 12 and (α, γ) (0.2, 0.1)), series lengths 50, 100, 250 and 500, 300 replicates per cell, smooth's default backcasting as a separate arm, failures counted.
Concepts 1
D32 statsforecast β cap: In statsforecast 2.1.1 (statsforecast.ets.initparam), when α is estimated its starting value is (0.20006 for ), and the upper bound of β is set to for the whole optimisation, so β cannot exceed .
Assumptions
The answer key states the findings that a correct paper on the brief would state, with correct numbers. Rested on by C1, C6, C7.
The agents under test, run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders. Rested on by C2, C3, C5.
Principal claims
Every lab record that the agents under test kept with claimset-lab on the ETS estimation brief passes claimset lab verify (record verifies).
Scope: data: agent runs on one simulation study task; conditions: claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, records as left at the end of each lab run; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E1 (T1), artefacts A1 · Assumes: C21 · Limitations: L2, L5 · Addresses: 2408.06292
In both of its lab runs, the Sonnet agent ran every confirmatory test under a plan with a computed decision rule, written with its hypotheses before any run and approved by the human before the test (plans before data).
Scope: data: agent runs on one simulation study task; conditions: claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the two Sonnet lab runs; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E1 (T1.r1, T1.r2), artefacts A1, A2 · Assumes: C22 · Limitations: L1, L2, L5 · Addresses: Nosek et al. (2018)
Neither lab run of the Haiku agent planned its tests before seeing the data (plans before data): one run made no plan at all, and the other set its hypotheses and thresholds after three pilot runs.
Scope: data: agent runs on one simulation study task; conditions: claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the two Haiku lab runs; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E1 (T1.r3, T1.r4), artefacts A1, A2 · Assumes: C22 · Limitations: L1, L2, L5 · Addresses: Nosek et al. (2018) · Addresses: Kaplan and Irvin (2015)
Three of the four lab runs of the agents under test left statsforecast's damped-trend option unset, so that their statsforecast arm selected between a damped and an undamped trend, against the fixed model forms of the brief (model form fixed).
Scope: data: agent runs on one simulation study task; conditions: claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the statsforecast arm of each run's script; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E1 (T1), artefacts A1 · Limitations: L1, L5
Every claimset that the agents under test completed with claimset-submit from one lab record validates with no errors (claimset validates), at the fourth to sixth validator outcome (validator outcomes until valid).
Scope: data: agent runs on one simulation study task; conditions: one lab record (lab-sonnet-1) completed four times, claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the final file of each claimset run; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E2 (T1), artefacts A1, A2 · Assumes: C22 · Limitations: L2, L5, L6 · Addresses: 2408.06292
No claim of the four claimsets that the agents under test completed with claimset-submit is wrong against the answer key: each is correct or partly correct (claim correctness).
Scope: data: agent runs on one simulation study task; conditions: one lab record (lab-sonnet-1) completed four times, claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the 20 claims of the four claimsets; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E2 (T1), artefacts A6, A8 · Assumes: C21 · Limitations: L3, L4, L5 · Addresses: 2408.06292
None of the four claimsets that the agents under test completed with claimset-submit states as a claim any of the main findings of the answer key: how large the estimation bias is, that statsforecast's optimiser fails to converge for the seasonal model, or that statsforecast caps β (answer-key findings stated).
Scope: data: agent runs on one simulation study task; conditions: one lab record (lab-sonnet-1) completed four times, claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, claims only, not limitations or open questions; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E2 (T1), artefacts A6, A8 · Uses: C12 · Assumes: C21 · Limitations: L3, L4, L5 · Addresses: 2408.06292
The agents under test answered the brief's questions about how large the bias and RMSE are with pass-or-fail tests against thresholds they chose themselves, so the estimates never became claims: claimset-lab turns only hypotheses decided by a rule into claims.
Scope: data: agent runs on one simulation study task; conditions: claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the three lab runs with a plan and the one without; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E1 (T2), artefacts A1 · Argument: W1 · Uses: C7, C12 · Limitations: L1, L5 · Addresses: Cumming (2014) · Addresses: Scheel et al. (2021)
The two claimsets that the Haiku agent completed give a scope to none of their three principal claims, against all three in each of the two that the Sonnet agent completed (principal claims with scope).
Scope: data: agent runs on one simulation study task; conditions: one lab record (lab-sonnet-1) completed four times, claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, principal claims only; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E2 (T1), artefacts A1 · Limitations: L1, L2, L5
Under EntryLib rubric 0.2, the one claimset that tested nothing under a plan gets a higher reliability score than each of the six claimsets whose principal claims were planned tests.
Scope: conditions: entrylib-rubric 0.2.0, without --novelty, the six claimsets of this study, claimsets written in either phase; not for: other rubric versions · Data: E2 (T1, T2), artefacts A8, A10 · Uses: C14 · Limitations: L5 · Addresses: Smaldino and McElreath (2016)
Under EntryLib rubric 0.2, a claimset that states no limitations and gives its claims no scope (claimset-haiku-2) gets a higher reliability score than one written from the same lab record with scoped claims and three specific limitations (claimset-sonnet-1).
Scope: conditions: entrylib-rubric 0.2.0, without --novelty, the six claimsets of this study, claimsets completed from one lab record; not for: other rubric versions · Data: E2 (T1.r1, T1.r4), artefacts A8, A10 · Uses: C13 · Limitations: L5 · Addresses: Smaldino and McElreath (2016)
Supporting claims
The lab record that the claimset phase started from holds three hypotheses, each a threshold test on the RMSE (a band for the ratio of the two packages, backcasting better in most cells at the shortest length, the longest length against the shortest), and no hypothesis about the size of the bias or the RMSE.
Scope: conditions: the lab record of run lab-sonnet-1, 1,130 events · Data: E2, artefacts A1 · Used by: C7, C8
In EntryLib rubric 0.2, an unmitigated limitation that could bias a claim's magnitude costs the claim 12 points, more than a missing scope (10 points).
Scope: conditions: entrylib-rubric 0.2.0, without --novelty, the six claimsets of this study, costs per claim; not for: other rubric versions · Data: E4 (T1.r1, T1.r2), artefacts A10 · Used by: C11
In EntryLib rubric 0.2, a descriptive claim made without a plan costs nothing in the plan group, and a planned claim whose plan is anchored only by a git push costs 3 points.
Scope: conditions: entrylib-rubric 0.2.0, without --novelty, the six claimsets of this study, claims of type descriptive; not for: other rubric versions · Data: E4 (T1.r3, T1.r4), artefacts A10 · Used by: C10
None of the four lab runs of the agents under test checked whether statsforecast's optimiser converged (convergence checked).
Scope: data: agent runs on one simulation study task; conditions: claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the statsforecast arm of each run's script; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E1 (T1), artefacts A1 · Uses: C16 · Limitations: L5
In the reference simulation, statsforecast's optimiser stopped at its iteration cap in nearly every ETS(A,N,A) fit, at every series length (statsforecast iteration cap).
Scope: conditions: statsforecast 2.1.1 AutoETS defaults, ETS(A,N,A), m = 12, α = 0.2, γ = 0.1 · Simulation: E3 (T2), artefacts A6, A7 · Used by: C15
In one of its two lab runs, the Haiku agent recorded a run with the human's actor ID as its author, and claimset lab accepted it (runs written as the human).
Scope: data: agent runs on one simulation study task; conditions: claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the two Haiku lab runs; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E1 (T1.r3), artefacts A1, A2 · Limitations: L1
In one of its two claimset runs, the Haiku agent replaced the model of the agent that had done the research, as the lab record names it, with its own (research agent's model kept).
Scope: data: agent runs on one simulation study task; conditions: one lab record (lab-sonnet-1) completed four times, claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the two Haiku claimset runs; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E2 (T1.r3), artefacts A1 · Limitations: L1
When statsforecast 2.1.1 fits ETS(A,A,N) with its defaults, its estimate of β never exceeds its starting value of α (statsforecast β cap), so that no estimate can lie above a true β of 0.2.
Scope: conditions: statsforecast 2.1.1 AutoETS defaults, model AAN, m = 1, ETS(A,A,N) with (α, β) = (0.5, 0.2), series lengths 50 to 500, 300 series each · Simulation: E3 (T1), artefacts A6, A7 · Used by: C20
The statsforecast β cap was reported by a Haiku agent lab run (lab-haiku-1, in an exploratory run) and was missing from the answer key as written before the lab phase.
Scope: data: agent runs on one simulation study task; conditions: claimset 0.41.6 with its skills read as files, Claude Code 2.1.322 Agent-tool subagents, two runs per agent, fixed answers in place of the human, the lab phase; not for: agents in sessions of their own with the guard hook active, other study tasks and fields · Data: E1 (T1.r3), artefacts A2, A6 · Uses: C19 · Limitations: L4
Evidence
The lab phase: two runs each of the Sonnet agent and the Haiku agent did the ETS estimation brief with claimset-lab. For each run the orchestrating agent read the final lab record (tools/lab_summary.py: plans, approvals and runs in record order, the actor of every event, anchors), the run's script, and its transcript (tools/transcript.py: every tool call, every refusal, model-call usage), and recorded the variables of table T1. Every question and answer is in log/questions.md.
Setup: studied Sonnet agent, Haiku agent · cases ETS estimation brief · procedure lab phase, fixed answers, claimset-lab · lens answer key, record verifies, tool refusals, plans before data, model form fixed, convergence checked, output tokens, active time, record anchored, runs written as the human, decision thresholds, hypotheses about an estimate
| Row | run | agent | record verifies | tool refusals | plans before data | model form fixed | convergence checked | record anchored | runs written as the human | output tokens | active time |
|---|---|---|---|---|---|---|---|---|---|---|---|
| r1 | lab-sonnet-1 | Sonnet agent | yes | 0 | all | yes | no | yes | 0 | 52609 | 11.2 |
| r2 | lab-sonnet-2 | Sonnet agent | yes | 0 | all | no | no | yes | 0 | 80054 | 13.5 |
| r3 | lab-haiku-1 | Haiku agent | yes | 3 | none | no | no | yes | 1 | 265066 | 16.7 |
| r4 | lab-haiku-2 | Haiku agent | yes | 1 | all, hypotheses after pilots | no | no | no | 0 | 122979 | 11.8 |
| Row | run | agent | decision thresholds | hypotheses about an estimate |
|---|---|---|---|---|
| r1 | lab-sonnet-1 | Sonnet agent | RMSE ratio of the two optimised arms between 0.9 and 1.1 in all 32 cells; backcasting with the lower RMSE in at least 6 of 8 cells at length 50; RMSE at length 500 below length 50 in all 24 comparisons | 0 |
| r2 | lab-sonnet-2 | Sonnet agent | RMSE at length 500 below length 50 in all 24 series; smooth with the lower RMSE than statsforecast in at least 75 % of 32 comparisons; backcasting below optimised initial states in at least 75 %; |bias| + 1.645 SE below 0.02 at length 500 for every parameter | 0 |
| r3 | lab-haiku-1 | Haiku agent | none | 3 |
| r4 | lab-haiku-2 | Haiku agent | no violations of: RMSE falling with length; statsforecast's RMSE within 10 % of smooth's; |bias| at most 0.02 at length 500 (not supported only at 12, 16 and 12 violations) | 0 |
The claimset phase: two runs each of the Sonnet agent and the Haiku agent exported one lab record and completed the claimset with claimset-submit. For each final claimset the orchestrating agent ran claimset validate and claimset score (EntryLib rubric 0.2), counted the validator outcomes in the transcript (tools/validate_runs.py), checked that no record-derived field (hypothesis, planning, planned_at, test, exploration counts) had changed, and scored every claim against the answer key (table T1). Table T2 scores, under the same rubric, the two claimsets that lab-phase agents wrote on their own records after finishing them (beyond their task).
Setup: studied Sonnet agent, Haiku agent · cases ETS estimation brief · procedure claimset phase, fixed answers, claimset-submit · lens answer key, EntryLib rubric 0.2, claimset validates, validator outcomes until valid, warnings left, reliability score, claim correctness, answer-key findings stated, principal claims with scope, limitations, research agent's model kept, interventions, output tokens
| Row | run | agent | claimset validates | validator outcomes until valid | warnings left | reliability score | claim correctness (outcome: correct) | claim correctness (outcome: partly correct) | claim correctness (outcome: wrong) | answer-key findings stated | principal claims with scope | limitations | research agent's model kept | interventions | output tokens |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| r1 | claimset-sonnet-1 | Sonnet agent | yes | 4 | 2 | 73 | 4 | 1 | 0 | 0 | 3 | 3 | yes | 0 | 50493 |
| r2 | claimset-sonnet-2 | Sonnet agent | yes | 5 | 3 | 72 | 5 | 0 | 0 | 0 | 3 | 3 | yes | 0 | 44792 |
| r3 | claimset-haiku-1 | Haiku agent | yes | 6 | 1 | 79 | 4 | 1 | 0 | 0 | 0 | 3 | no | 0 | 69641 |
| r4 | claimset-haiku-2 | Haiku agent | yes | 4 | 11 | 78 | 3 | 2 | 0 | 0 | 0 | 0 | yes | 1 | 55371 |
| Row | run | agent | planning of the principal claims | reliability score | limitations | principal claims with scope |
|---|---|---|---|---|---|---|
| r1 | lab-sonnet-2 | Sonnet agent | confirmatory | 73 | 4 | 3 |
| r2 | lab-haiku-1 | Haiku agent | exploratory | 92 | 0 | 3 |
The reference simulation of the brief (R = 300 per cell, 20 cells, three arms, 110 s on 4 cores), with optcheck.py and damped_check.py. Table T1: statsforecast's β estimates for ETS(A,A,N) with (α, β) = (0.5, 0.2), against smooth's; table T2: statsforecast fits of ETS(A,N,A) that stopped at the optimiser's iteration cap. The β cap was first reported by run lab-haiku-1 (its exploratory run RN1); it was verified here against the statsforecast 2.1.1 source and these estimates, and added to the answer key.
Setup: studied reference simulation · cases ETS estimation brief · procedure reference simulation · lens statsforecast β cap, statsforecast iteration cap
| Row | series length | largest statsforecast β estimate | statsforecast fits within 0.001 of 0.20006 | smooth fits with β above 0.20006 |
|---|---|---|---|---|
| r1 | 50 | 0.2001 | 100 | 131 |
| r2 | 100 | 0.2001 | 135 | 146 |
| r3 | 250 | 0.2001 | 132 | 132 |
| r4 | 500 | 0.2001 | 144 | 137 |
| Row | series length | statsforecast iteration cap |
|---|---|---|
| r1 | 50 | 288 |
| r2 | 100 | 296 |
| r3 | 250 | 300 |
| r4 | 500 | 299 |
A reading of the rubric's code: the costs EntryLib rubric 0.2 deducts for the problems that decide the scores compared in this paper.
Setup: studied EntryLib rubric 0.2 · cases EntryLib rubric 0.2 · procedure reading of the rubric's code · lens rubric cost
| Row | problem | rubric cost |
|---|---|---|
| r1 | a limitation that could bias magnitude, not mitigated | 12 |
| r2 | no scope | 10 |
| r3 | a plan anchored only by a git push | 3 |
| r4 | a descriptive claim without a plan | 0 |
The claimset-lab skill offers one route from a run to a claim: a hypothesis decided by a rule, with computed rules preferred whenever the outcome can be computed from numbers (SKILL.md, lines 83–100), and export makes claims only from hypotheses with a verdict or a final status (lab record v0.21 §9). Every planned run accordingly chose thresholds the brief did not give (an RMSE ratio within 10 %, |bias| at most 0.02, shares of at least 0.75, at least 12 violations), and its claims are the verdicts on them; the one run without a plan wrote notes and claims about the estimates themselves (the γ pile-up, statsforecast's seasonal α bias, the β cap).
Limitations
Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs.
The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine.
One scorer, the orchestrating agent, wrote the answer key and the protocol and scored every claim, knowing which agent wrote it.
The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key.
One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322).
One claimset run (claimset-haiku-2) stopped with its own stated work undone and was sent one message with no content ("Continue.") before it finished.
Open questions
Whether the differences between the Sonnet and Haiku agents hold over more runs, other study tasks and agents in sessions of their own is not established.
Whether refusing writes by a human actor in agent harnesses, and keeping the research agent's model through export, would prevent the misattributions of a run and of a study's research is not tested.
Whether a skill instruction to report each fit's model form and optimiser status as results would close the gaps in fixing the model form and checking convergence is not tested.
Whether a plan that fixes estimates rather than a decision rule would lead agents to state the brief's findings as claims is not tested.
References
- Nosek et al. (2018) Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. doi:10.1073/pnas.1708274114 Used by: Context
- Kaplan and Irvin (2015) Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. PLOS ONE, 10(8), e0132382. doi:10.1371/journal.pone.0132382 Used by: Context
- Scheel et al. (2021) Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2). doi:10.1177/25152459211007467 Used by: Context
- Cumming (2014) Cumming, G. (2014). The New Statistics. Psychological Science, 25(1), 7–29. doi:10.1177/0956797613504966 Used by: Context
- Smaldino and McElreath (2016) Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3(9), 160384. doi:10.1098/rsos.160384 Used by: Context
- 2408.06292 2408.06292 Used by: Context
Artefacts
- A1 · data The eight runs as the agents left them: each folder's files without its git directory, and its full git history (study branch included) in repo.bundle. runs
- A2 · trace The eight agents' transcripts (Claude Code subagent JSONL), with every tool call and result and the usage of every model call. transcripts
- A3 · prompt The task prompts given to the agents in each phase. prompts.md
- A4 · protocol The study brief and the fixed answers given in place of the human. study
- A5 · data Setup notes, every question and answer, agent IDs and usage per turn. log
- A6 · data The reference analysis: estimates.csv, summary.csv, consistency.csv, optcheck.csv, damped_check.csv and answer_key.md. reference
- A7 · code The reference analysis scripts simulate.py, optcheck.py and damped_check.py. reference
- A8 · data Scoring per run, rubric scores as JSON, summary.md and report.md. results
- A9 · code The scoring scripts transcript.py, lab_summary.py and validate_runs.py. tools
- A10 · software EntryLib's scoring rubric, entrylib-rubric 0.2.0 (src/rubric/rubric.json holds the costs; src/rubric/score.py applies them). 8a6f9f948a83c0a6f8c60c1e101ed5c4f5399caf
Score
Support of the paper: 76, Partly supported
Scores are computed by rules from the claimset file (EntryLib rubric 0.3): they check the record, not the science, and may change as the rubric and reviews develop. The rubric. Every issue as text, for an agent preparing the next version.
The mean of its 11 principal claims; the lowest is C9 at 66. Open problems in the principal claims and what they rest on, each counted once, with the most it costs a claim:
- −10 Open problems Limitation L1 (could reverse, not mitigated): Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs. In C2, C3, C4, C8, C9, C17, C18. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −10 Support Rests on evidence named without the observations the claim reports. In C1, C4, C5, C6, C7, C8, C9, C10, C12, C15, C16, C19, C20. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −5 Plan A principal claim reported without a plan recorded beforehand. In C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −5 Context A principal claim related to no earlier work. In C4, C9. To remove it: Relate the claim to the earlier work it extends, addresses or departs from.
- −6 Open problems Limitation L2 (could bias magnitude, not mitigated): The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine. In C1, C2, C3, C5, C9. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −3 Open problems Assumption C21 is not checked: The answer key states the findings that a correct paper on the brief would state, with correct numbers. In C1, C6, C7, C8. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Limitation L3 (could bias magnitude, mitigation partial): One scorer, the orchestrating agent, wrote the answer key and the protocol and scored every claim, knowing which agent wrote it. In C6, C7, C8. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −6 Open problems Limitation L4 (could bias magnitude, not mitigated): The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key. In C6, C7, C8, C20. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). In C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C15. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −3 Open problems Limitation L6 (could bias magnitude, not mitigated): One claimset run (claimset-haiku-2) stopped with its own stated work undone and was sent one message with no content ("Continue.") before it finished. In C5. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −3 Open problems Assumption C22 is not checked: The agents under test, run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders. In C2, C3, C5. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
Claims
C1: 75, Partly supported
Novelty: new, by the authors
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −6 Open problems Limitation L2 (could bias magnitude, not mitigated): The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Assumption C21 is not checked: The answer key states the findings that a correct paper on the brief would state, with correct numbers. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C2: 78, Partly supported
Novelty: new, by the authors
- −10 Open problems Limitation L1 (could reverse, not mitigated): Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Assumption C22 is not checked: The agents under test, run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −3 Open problems Limitation L2 (could bias magnitude, not mitigated): The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C3: 78, Partly supported
Novelty: new, by the authors
- −10 Open problems Limitation L1 (could reverse, not mitigated): Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Assumption C22 is not checked: The agents under test, run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −3 Open problems Limitation L2 (could bias magnitude, not mitigated): The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C4: 69, Partly supported
Novelty: new, by the authors
- −10 Open problems Limitation L1 (could reverse, not mitigated): Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −5 Context A principal claim related to no earlier work. To remove it: Relate the claim to the earlier work it extends, addresses or departs from.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C5: 72, Partly supported
Novelty: new, by the authors
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −6 Open problems Limitation L2 (could bias magnitude, not mitigated): The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Limitation L6 (could bias magnitude, not mitigated): One claimset run (claimset-haiku-2) stopped with its own stated work undone and was sent one message with no content ("Continue.") before it finished. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −3 Open problems Assumption C22 is not checked: The agents under test, run as subagents of an orchestrating session, act as they would in sessions of their own started in their run folders. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C6: 74, Partly supported
Novelty: new, by the authors
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −6 Open problems Limitation L4 (could bias magnitude, not mitigated): The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Assumption C21 is not checked: The answer key states the findings that a correct paper on the brief would state, with correct numbers. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Limitation L3 (could bias magnitude, mitigation partial): One scorer, the orchestrating agent, wrote the answer key and the protocol and scored every claim, knowing which agent wrote it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C7: 74, Partly supported
Novelty: new, by the authors
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −6 Open problems Limitation L4 (could bias magnitude, not mitigated): The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Assumption C21 is not checked: The answer key states the findings that a correct paper on the brief would state, with correct numbers. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −1 Open problems Limitation L3 (could bias magnitude, mitigation partial): One scorer, the orchestrating agent, wrote the answer key and the protocol and scored every claim, knowing which agent wrote it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C8: 67, Partly supported
Novelty: new, by the authors
- −10 Open problems Limitation L1 (could reverse, not mitigated): Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −3 Open problems Assumption C21 is not checked: The answer key states the findings that a correct paper on the brief would state, with correct numbers. To remove it: Test the assumption, or analyse how sensitive the claim is to it.
- −3 Open problems Limitation L4 (could bias magnitude, not mitigated): The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L3 (could bias magnitude, mitigation partial): One scorer, the orchestrating agent, wrote the answer key and the protocol and scored every claim, knowing which agent wrote it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C9: 66, Partly supported
Novelty: new, by the authors
- −10 Open problems Limitation L1 (could reverse, not mitigated): Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −5 Context A principal claim related to no earlier work. To remove it: Relate the claim to the earlier work it extends, addresses or departs from.
- −3 Open problems Limitation L2 (could bias magnitude, not mitigated): The agents ran as subagents of the orchestrating session: they read the skills as files, the claimset guard hook of their folders was not active, and they received the orchestrating session's own project instructions (which led one Haiku run to ask its questions one at a time); four runs shared one 4-core machine. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C10: 84, Well supported
Novelty: new, by the authors
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C11: 94, Well supported
Novelty: new, by the authors
- −5 Plan A principal claim reported without a plan recorded beforehand. To remove it: Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C12: 90, Well supported
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
C13: 100, Well supported
Nothing lost.
C14: 100, Well supported
Nothing lost.
C15: 89, Well supported
Novelty: new, by the authors
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −1 Open problems Limitation L5 (reduces transferability, not mitigated): One study task (an ETS simulation comparison in Python), one version of claimset (0.41.6) and of its skills, one harness (Claude Code 2.1.322). To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C16: 90, Well supported
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
C17: 90, Well supported
Novelty: new, by the authors
- −10 Open problems Limitation L1 (could reverse, not mitigated): Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C18: 90, Well supported
Novelty: new, by the authors
- −10 Open problems Limitation L1 (could reverse, not mitigated): Two runs per agent and phase: differences between the Sonnet and Haiku agents rest on two runs each and could reverse with more runs. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
C19: 90, Well supported
Novelty: new, by the authors
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
C20: 84, Well supported
- −10 Support Rests on evidence named without the observations the claim reports. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked.
- −6 Open problems Limitation L4 (could bias magnitude, not mitigated): The answer key was amended once during the lab phase, after a Haiku run reported statsforecast's β cap; the claimsets were scored against the amended key. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
Versions
- Version 2 · 12 November 2026 at 11:43 UTC · updated
- Version 1 (this page) · 12 November 2026 at 11:23 UTC · active
Every version is kept in the public registry (github.com/entrylib/registry).