A report may compare a college against another across many outcomes and highlight the one difference that passes a statistical threshold. If many comparisons were examined, that result needs to be interpreted in the context of the whole set. A threshold for one test does not automatically describe the chance of seeing at least one threshold crossing across many tests.

NIST describes methods for multiple comparisons, including methods designed for different sets of comparisons. This article uses a separate fictional calculation to show the issue, not an analysis of actual college data. The arithmetic below assumes valid tests under an explicit model; real studies require their own design and method.

Start with twenty hypothetical tests

Suppose an analyst tests twenty unrelated fictional outcomes. In the model, every null hypothesis is true, each test has a 5% chance of crossing its threshold under that null, and the twenty test outcomes are independent. These are assumptions, not a claim about actual educational outcomes.

Number of tests Probability of no false threshold crossings Probability of at least one
1 0.95 5.0%
5 0.95⁵ ≈ 0.7738 About 22.6%
20 0.95²⁰ ≈ 0.3585 About 64.2%

Under the stipulated model, the probability of at least one crossing is 1 − 0.95²⁰, approximately 64.2%. The expected number of crossings is 20 × 0.05 = 1. These are different quantities: an expected count of one does not mean every set contains exactly one crossing.

The calculation does not say that a particular observed result has a 64.2% probability of being false. It describes the chance of at least one crossing across the full set when all twenty null hypotheses are true and the model's independence and test assumptions hold.

Record the entire comparison set

List which outcomes, subgroups, time periods and institutions were examined, including results not highlighted. A comparison selected after inspecting many possibilities is different from one specified before the data were examined. If only a selected result is published, ask how many other analyses were considered and whether selection was accounted for.

A nominal p-value alone does not measure the size or practical importance of a difference. Report the estimated difference, uncertainty, population, data period and outcome definition. A large sample can produce a small p-value for a small difference, while an imprecise result can leave a meaningful effect unresolved.

Consider a suitable family-level method

One simple illustration is to divide a desired overall threshold of 0.05 by twenty, yielding 0.0025 per test. Under appropriate valid individual tests, the Bonferroni bound controls the chance of one or more false rejections at no more than 0.05 for that defined family, including when the tests are dependent. This is a conservative bound, not a guarantee that the tests' assumptions or data are correct.

Which comparisons belong in the family depends on the research question and analysis plan. Other methods target different error criteria or comparison structures. The appropriate choice requires statistical judgment; multiplying or dividing p-values without understanding the design is not a substitute. Dependence also means the independent 64.2% illustration above cannot be transferred unchanged to related outcomes.

Distinguish exploration from confirmation

A pattern discovered while exploring data can motivate a prespecified analysis in an independent dataset or later period. Such confirmation still needs comparable definitions, suitable controls and a transparent analysis plan. Failing to cross a threshold does not establish that an effect is exactly zero, and crossing one does not prove a causal explanation.

When comparing colleges, check whether the report shows all outcomes or only favorable highlights. A useful evidence note records the comparison family, selection process, effect sizes and uncertainty. This makes it possible to assess an interesting result without mistaking a search across many tests for a single preplanned test.