Experiments and inference

Version 0.2 (draft, to be reviewed by someone who works in the field) · 2026-11-13

For studies that compare conditions, methods or groups, or estimate an effect: designed experiments, simulation studies, observational comparisons and causal questions.

Principle

A conclusion rests on what is estimated and how the comparison was decided. Say first what quantity the claim is about (the estimand: which effect, on whom, compared with what), then choose the design and the analysis that estimate it, and fix both before the data are seen (Lundberg, Johnson and Stewart 2021; Nosek et al. 2018). The analysis answers the question the design was built for, not the one that turned out convenient.

estimand

When: any comparison or effect.
Default: state the estimand before the design: the population or cases, the conditions compared, the outcome, how events after assignment that change or end the outcome are handled (stopping or switching a treatment, a unit that drops out of the process studied), and the summary measure (a difference in means, a ratio, a share). This is the estimands framework of the ICH E9(R1) addendum for clinical trials (Kahan et al. 2024); outside trials, the same questions define the target quantity (Lundberg, Johnson and Stewart 2021). Missing data are a matter of how the estimand is estimated, not of what it is.
Why: a precise estimand connects the statistical result to the claim it supports; without it, the same data support several different claims, and the analysis can be chosen after the fact (Lundberg, Johnson and Stewart 2021; Nosek et al. 2018).
References: Lundberg, Johnson and Stewart 2021; Kahan et al. 2024; Nosek et al. 2018.

effect-sizes

When: reporting any comparison.
Default: report the size of the effect with its interval, not only whether a test rejected (Cumming 2014).
Why: a p-value does not measure the size of an effect or the importance of a result (Wasserstein and Lazar 2016); an estimate with its interval shows how large the effect may be, against which its importance can be judged (Cumming 2014).
Avoid: "significant" or "not significant" as the finding (Wasserstein, Schirm and Lazar 2019).
References: Wasserstein and Lazar 2016; Wasserstein, Schirm and Lazar 2019; Cumming 2014.

no-difference

When: the claim is that there is no difference, or no meaningful one.
Default: an equivalence test against bounds fixed in advance: the smallest difference that would matter (Lakens 2017).
Why: failing to reject a null hypothesis is not evidence of no difference (Altman and Bland 1995; Lakens 2017).
References: Lakens 2017; Altman and Bland 1995.

multiple-tests

When: several tests decide the claims of one study.
Default: fix the correction in advance. Use Holm's procedure to control the family-wise error rate when each claim must stand on its own (Holm 1979); it holds whatever the dependence between the tests. Use Benjamini–Hochberg to control the false discovery rate when many tests screen for effects (Benjamini and Hochberg 1995); it is proved for independent or positively dependent tests (Benjamini and Yekutieli 2001), and under arbitrary dependence the Benjamini–Yekutieli correction is needed (Benjamini and Yekutieli 2001).
Why: without a correction, the chance of a false claim grows with the number of tests; a correction chosen after the results are seen can be chosen to fit them (Nosek et al. 2018).
References: Holm 1979; Benjamini and Hochberg 1995; Benjamini and Yekutieli 2001.

simulation-studies

When: methods are evaluated on simulated data.
Default: plan and report by aims, data-generating mechanisms, estimands, methods and performance measures (ADEMP); choose the number of repetitions for the Monte Carlo standard error you can accept, and report that error with each estimated performance measure (Morris, White and Crowther 2019).
References: Morris, White and Crowther 2019.

preregistration

When: any test meant to be confirmatory.
Default: fix the research question, the hypotheses, the design and the analysis plan before the data are seen, in a record with a trustworthy time (in a claimset lab, the plan) (Nosek et al. 2018).
Why: a test is a prediction only when it was stated before the outcomes were seen; the same analysis chosen afterwards is a postdiction (Nosek et al. 2018).
References: Nosek et al. 2018.

reporting

Default: follow the reporting guideline for the design: CONSORT 2025 for randomised trials (Hopewell et al. 2025), STROBE for observational studies in epidemiology (cohort, case-control and cross-sectional) (von Elm et al. 2007), and others at the EQUATOR Network (https://www.equator-network.org). Simulation studies report by ADEMP (Morris, White and Crowther 2019).
References: Hopewell et al. 2025; von Elm et al. 2007; Morris, White and Crowther 2019.

precedent

When: a test or a level is proposed because others used it.
Default: decide by the estimand and the design; a precedent says what others did, not why it fits this study. Justify each choice, the significance level included (Lakens et al. 2018).
References: Lakens et al. 2018.

References

Altman, D. G., and Bland, J. M. (1995). Statistics notes: absence of evidence is not evidence of absence. BMJ, 311(7003), 485. doi:10.1136/bmj.311.7003.485
Benjamini, Y., and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B, 57(1), 289–300. doi:10.1111/j.2517-6161.1995.tb02031.x
Benjamini, Y., and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29(4), 1165–1188. doi:10.1214/aos/1013699998
Cumming, G. (2014). The new statistics: why and how. Psychological Science, 25(1), 7–29. doi:10.1177/0956797613504966
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., and Vandenbroucke, J. P. (2007). The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. The Lancet, 370(9596), 1453–1457. doi:10.1016/S0140-6736(07)61602-X
Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. https://www.jstor.org/stable/4615733
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., et al. (2025). CONSORT 2025 statement: updated guideline for reporting randomised trials. BMJ, 389, e081123. doi:10.1136/bmj-2024-081123
Kahan, B. C., Hindley, J., Edwards, M., Cro, S., and Morris, T. P. (2024). The estimands framework: a primer on the ICH E9(R1) addendum. BMJ, 384, e076316. doi:10.1136/bmj-2023-076316
Lakens, D. (2017). Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362. doi:10.1177/1948550617697177
Lakens, D., Adolfi, F. G., Albers, C. J., et al. (2018). Justify your alpha. Nature Human Behaviour, 2(3), 168–171. doi:10.1038/s41562-018-0311-x
Lundberg, I., Johnson, R., and Stewart, B. M. (2021). What is your estimand? Defining the target quantity connects statistical evidence to theory. American Sociological Review, 86(3), 532–565. doi:10.1177/00031224211004187
Morris, T. P., White, I. R., and Crowther, M. J. (2019). Using simulation studies to evaluate statistical methods. Statistics in Medicine, 38(11), 2074–2102. doi:10.1002/sim.8086
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., and Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. doi:10.1073/pnas.1708274114
Wasserstein, R. L., and Lazar, N. A. (2016). The ASA's statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129–133. doi:10.1080/00031305.2016.1154108
Wasserstein, R. L., Schirm, A. L., and Lazar, N. A. (2019). Moving to a world beyond "p < 0.05". The American Statistician, 73(sup1), 1–19. doi:10.1080/00031305.2019.1583913

Versions

0.1 (2026-11-13): first draft, for review by someone in the field.
0.2 (2026-11-13): each citation checked against the paper. The estimand entry follows the estimands framework (Lundberg, Johnson and Stewart 2021; Kahan et al. 2024) instead of citing a paper on simulation studies; CONSORT 2025 replaces CONSORT 2010; Benjamini–Hochberg states its dependence condition; the effect-size, no-difference, multiple-tests, preregistration and precedent entries gain the references for what they say.

From entrylib/study-planning at 104ba1c (2026-11-13), as the installed claimset package carries it. Changes are proposed there, with the references that support them.