claimset-lab
Research with your agent that anyone can check
claimset-lab keeps a record of your research while it happens: what you asked, what you planned before you saw any data, every run of code with its inputs and outputs, every decision and every result, including the ones that did not work. Your agent writes it as it works; at the end it becomes a claimset whose claims trace back to the record.
The problem
An agent can run a hundred analyses in an afternoon and report the one that worked. Nothing in the finished paper shows whether a test was planned or chosen after the fact, how many attempts came before the reported one, or whether a number came from code or from memory. Readers, and the models that read for them, have to take it on trust.
claimset-lab makes those things part of the record, so they can be checked instead of trusted.
What the record makes checkable
Planned before tested
A test counts as confirmatory only if its plan, with the significance level, sample and the rule for each outcome, was recorded and pushed before the run. Anything else is reported as exploration.
Every number from code
Each run records its command, code commit, environment, seeds, parameters, input-file hashes and outputs. Results enter the record from the files the code wrote, never retyped.
Failures counted
Refuted and abandoned hypotheses stay in the record with their reasons. A planned test gets one attempt, so it cannot be rerun until it comes out right. Negative results are published like positive ones.
Hard to rewrite
The record is append-only and hash-chained, kept on its own git branch and pushed after every step. Corrections are new entries that point at what they replace; nothing is edited in place.
People decide
Every actor is registered: agents with their model, people with their ORCID iD. A supervisor's approval of a plan is recorded in their own words, with when they gave it; an agent never approves its own plan.
Checks by others
Another agent or a colleague can reproduce runs and re-derive results in their own log. Their verdicts, confirmed or disputed, travel with the claims they checked.
How it works
Your agent runs the commands as it works; you approve plans and answer its questions.
- Frame: questions, hypotheses with what they are about, the terms the study uses, the assumptions it makes.
- Plan: design, analysis and decision rule, fixed before the data is seen, and approved by the supervisor when there is one.
- Run and record: exploratory and confirmatory runs, deviations from the plan, decisions, and the verdict the plan's rule gives.
- Export and publish: the record becomes a claimset: hypotheses become claims, plans and runs their evidence. Submit it to EntryLib with the record attached.
From exploration to a planned test
| Your agent runs | The record gets |
|---|---|
claimset lab hypothesis "M helps most at horizons 7–12." --from-exploration --prompted-by RN1 |
H2, marked as suggested by RN1 |
claimset lab plan --kind confirmatory --hypotheses H2 --supported "p < alpha and d > 0" … |
PL1, pushed before any run |
claimset lab run --plan PL1 -- python3 confirm.py --holdout | RN2, with results RS2–RS3 |
claimset lab verdict PL1 | p = 0.004, d = 0.031: supported |
Beyond computation
The lab records any research that can be planned and logged: experiments, simulations, benchmarks, data analyses, and studies of documents and other sources. What one kind of work needs beyond that comes as an extension. The qualitative extension records sources as they were read, codes assigned to passages with the rules for each code, and counts that a planned test can use; the record then shows that the sampling was fixed before the coding, and the coding before the counting.
What readers see
A claimset from the lab is labelled lab and links to its record. Each claim says whether it was planned, under which plan and runs, and what deviated from the plan. Readers and models can tell a tested prediction from a pattern found by looking.
Get started
claimset-lab is a command-line tool and a set of instructions for research agents. It works with any agent that can run commands in your project; it needs Python 3.10 or later and git, and nothing else. Free and open source.
- Install the tool
pip install claimset - Give your agent the instructions
claimset skill install - Ask your agent to start a study
"Start a lab record for this study and keep it as we work."