CLAIM C21 of entrylib:2026.000008 Evaluating forecasts and classifiers: which measures, scoring rules and tests to use licence CC-BY-4.0; contributions C21 extensions review 0.1 DEFINITIONS D25 [method] Friedman test: A non-parametric test of whether several methods differ, applied to their ranks on each of several data sets. [source B5] D26 [method] Nemenyi test: A post-hoc test after the Friedman test that compares every pair of methods by their mean ranks against a critical difference. [source B5] CLAIMS C21 [methodological; principal; synthesis; support: argument W21; uses C51; limitations L5; strength high (author); verifiability open; planning exploratory; outcome supported] [review: kind recommendation] To tell whether several methods whose aggregate accuracy is similar differ statistically significantly over many series or data sets, their ranks on each series should be tested with the [[D25:Friedman test]] followed by [[D26:Nemenyi test]] post-hoc comparisons. scope: data_type: comparisons of several methods over many series or data sets C51 [methodological; supporting; synthesis; support: cited B5, at Section 3.2.2 (Friedman test), cited B5, at Section 5 (Conclusion); limitations L5; verifiability open; planning exploratory; outcome supported] [review: extracts [{"reference": "B5", "locator": "Section 3.2.2 (Friedman test)", "lab_event": "XT24", "exact": "It ranks the algorithms for each data set separately", "prefix": "of the repeated-measures\nANOVA. ", "suffix": ", the best performing algorithm ", "start": 38209, "end": 38261, "artefact": "A4"}, {"reference": "B5", "locator": "Section 3.2.2 (Friedman test)", "lab_event": "XT25", "exact": "The Nemenyi test", "prefix": "n proceed with a post-hoc test. ", "suffix": " (Ne-\nmenyi, 1963) is similar to", "start": 40625, "end": 40641, "artefact": "A4"}, {"reference": "B5", "locator": "Section 5 (Conclusion)", "lab_event": "XT26", "exact": "They are appropriate since they assume some, but limited commensurability", "prefix": " are suitable for our\nproblems. ", "suffix": ". They\nare safer than parametric", "start": 432529, "end": 432602, "artefact": "A4"}]; kind summary] The [[D25:Friedman test]], followed by the [[D26:Nemenyi test]], compares several methods over many data sets by their ranks on each data set, assuming only limited commensurability of the measure across data sets. scope: data_type: comparisons of several methods over many data sets WARRANTS W21 [deductive; premises C51] The Friedman test, followed by the Nemenyi test, compares methods by their ranks on each data set and assumes only limited commensurability of the measure across data sets; it tells whether differences between methods whose aggregate accuracy is close are larger than chance. LIMITATIONS L5 [researcher_position; affects C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16, C17, C18, C19, C20, C21, C22, C23, C24, C25, C26, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C40, C41, C42, C43, C44, C45, C46, C47, C48, C49, C50, C51, C52, C53; could_bias_magnitude; mitigation none] The claims restate EntryLib's study-planning guide on prediction, drafted by an agent of the same system that checked them here, and were stated before the works were read; the works were read to confirm what the guide states, and no search was run for work that disagrees with it. ARTEFACTS A4 [data] Statistical comparisons of classifiers over multiple data sets (Demsar 2006, JMLR 7, 1-30) (application/pdf), saved by hand on 2026-11-13; recorded in the lab record as SR4 on 2026-11-13. venue: Journal of Machine Learning Research. Written by others. REFERENCES B5 https://jmlr.org/papers/v7/demsar06a.html (url) HOW TO CITE This claim: Svetunkov, I. (guarantor) (2026). Claim C21, in: Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2#C21. https://entrylib.org/p/2026.000008/C21 The claimset it is from: Svetunkov, I. (guarantor) (2026). Evaluating forecasts and classifiers: which measures, scoring rules and tests to use. EntryLib. entrylib:2026.000008@v2. https://entrylib.org/p/2026.000008@v2