Core ML

Statistical Model Comparison

Two accuracies from one test set aren't independent draws. McNemar's test uses only the disagreements, more powerful than treating the two scores as separate.

Two models scored on the same rows are compared by looking only at where they disagree, not by treating their two accuracy numbers as independent draws

Why Does This Exist?

Two models score 90.4% and 88.1% on the same test set. Is that a real difference, or one you'd expect from resampling noise alone? Model evaluation tells you what to measure. It stops short of telling you whether the gap between two measurements means anything, and eyeballing a leaderboard is where a lot of real research and product decisions quietly go wrong.

Here's the case we'll carry down the page. You've shipped model A at 90.4% accuracy on 1,000 held-out rows and a candidate model B scores 88.1% on the identical rows. Run the comparison as if the two accuracies came from separate, independent samples — a standard two-proportion test — and the difference looks statistically ambiguous. But A and B were scored on the same rows, which is a different, more informative kind of data than two independent samples, and using the wrong test on it either erases a real signal or invents one that isn't there.

Think of It Like This

Grading two chess players by their head-to-head games, not their season records

Two chess players have similar overall win rates against the field, and you want to know which one is actually stronger. Comparing their season win percentages as if they were two unrelated numbers throws away the one dataset that would actually answer the question: the games where they played each other.

Those head-to-head games carry all the real information about who's better — every other game, against different opponents of different strength, adds noise that has nothing to do with either player specifically. A fair judge ignores each player's overall record and looks only at the matches between the two of them.

McNemar's test is exactly that judge, applied to two models scored on the same rows: it discards every row both got right and every row both got wrong, because those carry no information about which model is better, and decides the comparison entirely from the rows where they disagreed.

How It Actually Works

Paired data changes which test is correct, not just which is more convenient

Two models evaluated on the same test set produce a paired comparison — each row has one outcome from A and one from B, not two separate populations. Treating it as unpaired (a two-proportion z-test on the two accuracy counts) discards the pairing entirely and estimates a larger variance than the comparison actually has, because it can't use the fact that most rows agree and carry zero information about which model is better. The correct test conditions on exactly that: McNemar's test looks only at the disagreement cells — rows where A was right and B was wrong, and rows where B was right and A was wrong — and asks whether that 2-by-2 split of disagreements is far from the 50/50 a coin flip would produce.

Why the paired test is so much more powerful

All the information distinguishing two models that agree on 95% of rows lives entirely in the 5% where they differ; the 95% they agree on is uninformative regardless of whether both are right or both are wrong. An unpaired test spreads its statistical power across the full sample including that uninformative 95%, diluting the signal. A paired test spends its power only where the signal actually is, which is why the same underlying difference can look inconclusive one way and highly significant the other — not because the paired test is more lenient, but because it's answering the comparison with the right amount of relevant data instead of the wrong amount.

Beyond binary accuracy: the rest of the family

McNemar covers paired binary outcomes — right or wrong, per row. When the comparison is over a continuous metric per fold or per row instead — five cross-validation fold scores for model A against the same five for model B — the Wilcoxon signed-rank test is the paired, distribution-free equivalent, ranking the per-fold differences rather than assuming they're normally distributed. A corrected paired t-test exists for the same setting under a normality assumption, but ordinary cross-validation folds share training data across folds and violate the independence a plain paired t-test needs; the correction adjusts the variance estimate for that shared-data effect specifically, rather than pretending the folds are as independent as fresh samples would be.

Show Me the Code

The same two models, same 1,000 rows, compared the wrong way and the right way.

1import numpy as np2from scipy import stats3
4rng = np.random.default_rng(6)5n = 10006truth = rng.integers(0, 2, n)7model_a = np.where(rng.random(n) < 0.90, truth, 1 - truth)  # 90% accurate baseline8model_b = model_a.copy()9disagree = rng.random(n) < 0.08  # B disagrees with A on 8% of rows, and loses most of them10model_b[disagree] = np.where(rng.random(int(disagree.sum())) < 0.70, truth[disagree], 1 - truth[disagree])11
12acc_a, acc_b = float((model_a == truth).mean()), float((model_b == truth).mean())13pooled = (acc_a + acc_b) / 214se = np.sqrt(pooled * (1 - pooled) * (2 / n))               # treats the two accuracies as independent15p_unpaired = float(2 * (1 - stats.norm.cdf(abs(acc_b - acc_a) / se)))16
17b_right = int(((model_b == truth) & (model_a != truth)).sum())  # only the rows where they disagree18a_right = int(((model_a == truth) & (model_b != truth)).sum())19p_mcnemar = float(stats.binomtest(b_right, b_right + a_right, 0.5).pvalue)20
21print(f"accuracy: A={acc_a:.3f}  B={acc_b:.3f}")22print(f"unpaired z-test p-value: {p_unpaired:.3f}   (looks inconclusive)")23print(f"McNemar p-value:         {p_mcnemar:.4f}   (disagreements: B-right {b_right}, A-right {a_right})")24# -> accuracy: A=0.904  B=0.88125# -> unpaired z-test p-value: 0.097   (looks inconclusive)26# -> McNemar p-value:         0.0000   (disagreements: B-right 3, A-right 26)

The unpaired test's p-value of 0.097 would normally be read as "not significant". McNemar's test on the exact same data, looking only at the 29 rows where the models actually disagreed, returns a p-value near zero — because A wins those disagreements 26 to 3, a lopsided split the unpaired test's wider, uninformed variance estimate couldn't see.

Watch Out For

Running a two-sample test on two models scored on the same rows

The single most common version of this mistake: two accuracy percentages go into a standard two-proportion test because that's the test everyone remembers, and the pairing information — that both numbers came from the identical set of rows — gets discarded without anyone deciding to discard it. The fix is a one-line question before choosing a test: were both models scored on the same examples? If yes, the comparison is paired, and McNemar (for binary outcomes) or Wilcoxon (for continuous per-fold scores) is the correct family, not a two-sample test.

Treating a significant McNemar result as license to skip nested validation

A significant paired test says the two models' disagreements on this test set are unlikely to be a coin flip — it says nothing about whether the model that won was itself chosen honestly, if its hyperparameters were tuned against the same data being used for the comparison. A statistically solid comparison built on top of an optimistically tuned model inherits that optimism regardless of how correctly the final comparison test was chosen. The two checks are independent and both required.

The Quick Version

Two models scored on the same test set produce paired data, not two independent samples — using an unpaired test throws away the pairing and estimates the wrong variance.
McNemar's test compares two models on binary outcomes by looking only at the rows where they disagree, since agreement carries no information about which model is better.
Paired tests are far more powerful than unpaired ones on the same underlying difference, because they spend statistical power only where the signal actually lives.
The Wilcoxon signed-rank test is the paired, distribution-free option for comparing continuous per-fold scores; a corrected paired t-test is the normal-theory option, adjusted for cross-validation folds sharing training data.
A correct comparison test and an honestly tuned model are separate requirements — passing one says nothing about the other.