The claim
To tell whether several methods whose aggregate accuracy is similar differ statistically significantly over many series or data sets, their ranks on each series should be tested with the Friedman test followed by Nemenyi test post-hoc comparisons.
Scope: data: comparisons of several methods over many series or data sets · Argument: W21 · Uses: C51 · Limitations: L5
Claims it rests on
The Friedman test, followed by the Nemenyi test, compares several methods over many data sets by their ranks on each data set, assuming only limited commensurability of the measure across data sets.
Scope: data: comparisons of several methods over many data sets · Cited: https://jmlr.org/papers/v7/demsar06a.html, Section 3.2.2 (Friedman test) · Cited: https://jmlr.org/papers/v7/demsar06a.html, Section 5 (Conclusion) · Limitations: L5 · Used by: C21
Evidence
The Friedman test, followed by the Nemenyi test, compares methods by their ranks on each data set and assumes only limited commensurability of the measure across data sets; it tells whether differences between methods whose aggregate accuracy is close are larger than chance.
Limitations
The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it.
Definitions
Methods 2
D25 Friedman test: A non-parametric test of whether several methods differ, applied to their ranks on each of several data sets. Source: https://jmlr.org/papers/v7/demsar06a.html.
D26 Nemenyi test: A post-hoc test after the Friedman test that compares every pair of methods by their mean ranks against a critical difference. Source: https://jmlr.org/papers/v7/demsar06a.html.
References
- https://jmlr.org/papers/v7/demsar06a.html Used by: C51, D25, D26, A4
Artefacts
- A4 · data · https://jmlr.org/papers/v7/demsar06a.html Statistical comparisons of classifiers over multiple data sets (Demsar 2006, JMLR 7, 1-30) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR4 on 2026-11-13. venue: Journal of Machine Learning Research. Written by others. demsar06a.pdf
Score
Scores are computed by rules from the claimset file (EntryLib rubric 0.7): they check the record, not the science, and may change as the rubric and reviews develop. The rubric.
C21: 81, Well supported
Novelty: summarises earlier work, by the authors
- −10 Evidence Rests on cited work only; cited work read in full, without the passage it rests on; an argument from other claims. To remove it: Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked; for cited work, read the work in full and quote the passage the claim rests on from a captured copy (claimset extension review).
- −6 Open problems Limitation L5 (could bias magnitude, not mitigated): The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.
- −3 Plan The review's search for the literature: none is recorded; the claims were confirmed against works the authors knew. To remove it: Record the protocol, then the searches and the screening of what they found (claimset lab protocol, search, screen), and export them.
C51: 94, Well supported
Novelty: summarises earlier work, by the authors
- −6 Open problems Limitation L5 (could bias magnitude, not mitigated): The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. To remove it: Mitigate it with evidence, or narrow the claim's scope so that it no longer applies.