A survey reports 72% for one institution and 75% for another. Ordering the displayed estimates is easy. Establishing that the underlying populations differ is harder, because survey estimates can vary with the sample. Before calling a three-point gap decisive, find the published uncertainty measures and the explanation of how they were calculated.

The Office for National Statistics explains sampling uncertainty and confidence intervals. Under repeated application of a valid 95% confidence-interval procedure, about 95% of the resulting intervals would contain the fixed population quantity. This does not mean that 95% of students fall inside the interval or that every error in the survey has been captured.

Start with the published intervals

Imagine two independent, simple-random-sample surveys measuring the same binary outcome at the same time. Each has 400 respondents. All percentages here are fictional. Institution A's estimate is 72%, and B's is 75%. A basic normal approximation gives a standard error of the proportion equal to the square root of p times one minus p divided by n.

Survey Estimate Approximate standard error Approximate 95% interval
A 72% 2.245 percentage points 67.6% to 76.4%
B 75% 2.165 percentage points 70.8% to 79.2%

For the difference B minus A, the independent-sample standard error is approximately 3.119 percentage points: the square root of the sum of the two squared standard errors. The estimated three-point difference therefore has an approximate 95% interval of −3.1 to 9.1 percentage points, using 1.96 standard errors on either side.

This illustration does not establish a positive underlying difference because zero lies within that interval. It also does not establish that the populations are identical. The available evidence is compatible with several differences in either direction. “No clear difference detected under this model” is more precise than “the colleges are the same.”

Do not apply this formula indiscriminately

Real education surveys can use weights, clustering, stratification or a near-census collection. Their uncertainty methods may differ substantially from this simple example. Use the publisher's design-based estimates or documented comparison tests rather than replacing them with a convenient formula and the raw response count.

Overlapping individual confidence intervals are not themselves a complete significance test for the difference. Dependence between estimates also matters. If the same people are measured twice, the independent-sample formula above is inappropriate. Identify the comparison the publisher actually supports, especially when a dashboard marks results as above or below a benchmark.

Sampling uncertainty is only one issue. Nonresponse, coverage gaps, ambiguous questions and inaccurate answers can bias results in ways the displayed interval does not describe. A narrow interval around a systematically biased estimate remains a narrow interval around a biased estimate. Check participation and methodology alongside precision.

Use uncertainty in the shortlist

Keep the reported estimate, interval, population definition and survey year together. A reader can still prefer an institution for curriculum or location when the survey gap is uncertain. The point is to avoid assigning a large decision weight to a small numerical separation that the evidence cannot resolve reliably.

Comparisons across many institutions also introduce a multiple-comparison problem: some apparent differences can arise by chance when many tests are examined.

If the publisher provides no suitable uncertainty measure, record that limitation instead of inventing one. Ask for documentation, and make the shortlist robust to plausible changes in the ordering. In the fictional comparison, a three-point headline does not justify a confident ranking. A transparent worksheet shows the gap and its limits, allowing stronger evidence elsewhere in the decision to carry appropriate weight.