Likelihood Ratio Tests

To compare two nested models, fit both and weigh the gap. Twice the log lift over the smaller model tells you whether the extra knobs earned it.

The constrained peak sits below the full peak, and twice the log gap at 4.03 clears the 3.84 chi-square cutoff to reject.

Why Does This Exist?

Does this feature help means comparing a small model against one with extra knobs. The big model always fits training data at least as well, so raw fit cannot decide. You need the price of flexibility.

The likelihood ratio pays exactly that price, with Wilks turning the gap into chi-square units. It extends hypothesis testing with the workhorse behind ANOVA, deviance, and many model checks, assuming the MLE machinery in maximum likelihood estimation. Interval reading of the same uncertainty lives in confidence intervals.

The prerequisite is one idea: log-likelihood $\ell(\theta)$ scores parameters by fit with $\theta$ the parameters, and its maximum is the MLE. The gap $\ell_1 - \ell_0 \ge 0$ measures what the extra knobs bought, and doubling turns that gap into chi-square units.

Think of It Like This

Two tailors, one surcharge

Two tailors fit you a suit. The standard tailor uses your size, and the bespoke tailor adds shoulder, drape, and cuff knobs and fits slightly better. The question is whether the better fit beats the bespoke surcharge.

Twice the log gap is the fit improvement in standard currency, and the chi-square cutoff is the surcharge for $d$ knobs, which is 3.84 for one knob at 5 percent. Clear the surcharge and the knobs earned it, otherwise keep the standard suit.

Where it stops: tailors can overfit fabric quirks and models can overfit noise, which is why the cutoff grows with $d$ . The analogy also hides nesting: the standard pattern must be a restricted bespoke pattern, or the surcharge table does not apply.

How It Actually Works

Fit the full estimate $\hat{\theta}$ and the constrained estimate $\hat{\theta}_0$ , where the hat marks maxima of the log-likelihood, then compute $D = 2(\ell(\hat{\theta}) - \ell(\hat{\theta}_0))$ with $\ell$ the log-likelihood. Reject the null $H_0$ when $D > \chi^2_{d, 1-\alpha}$ , where $d$ counts extra parameters and $\alpha$ is the error rate. Wilks says $D$ tends to $\chi^2_d$ under nesting, interior optima, and enough data.

Siblings agree asymptotically: Wald standardises the estimate distance over its standard error, and the score test uses the slope at $H_0$ . All three match on large samples, and the likelihood ratio respects reparameterisation best.

Worked example

One hundred flips land 60 heads, testing fair $p = 0.5$ . The full MLE is $\hat{p} = 0.6$ with $\ell_1 = 60\log 0.6 + 40\log 0.4 \approx -67.30$ . The null gives $\ell_0 = 100\log 0.5 \approx -69.31$ . The gap is $2.01$ , doubled to $D \approx 4.03 > 3.84$ , so reject fair at 5 percent by 0.19. A Wald check agrees: $(0.6 - 0.5)/0.05 = 2.0$ , borderline the same verdict with one extra parameter against the same 3.84 bar.

Code

The snippet recomputes the coin verdict with only the standard library.

1import math2
3
4k, n = 60, 1005ell1 = k * math.log(k / n) + (n - k) * math.log(1 - k / n)6ell0 = n * math.log(0.5)7print(round(ell1, 2))  # -> -67.38print(round(ell0, 2))  # -> -69.319print(round(2 * (ell1 - ell0), 2))  # -> 4.0310print(round((0.6 - 0.5) / 0.05, 2))  # -> 2.0

Doubled gap 4.03 clears 3.84 while Wald reports 2.0, the same data and the same verdict as the hand calculation.

Watch Out For

Testing non-nested models with Wilks

Comparing a Poisson against a negative binomial by a chi-square cutoff misfires when neither nests the other: the limit law is not chi-square and the cutoff is fiction. The symptom is wildly significant verdicts that vanish under cross-validation.

Use Vuong, information criteria, or held-out likelihood for non-nested choices. Reserve Wilks for true nesting where $H_0$ restricts $H_1$ .

Testing on the boundary

Testing variance equal to 0 or mixture weight equal to 0 parks $H_0$ on the parameter edge, halving the limit law toward a mixture such as one half chi-square zero plus one half chi-square one. Naive chi-square cutoffs run conservative and miss real effects.

Use boundary-corrected mixtures or bootstrap the null. When the null sits on an edge, distrust textbook cutoffs.

The Quick Version

The likelihood ratio test doubles the log-likelihood gap between nested fits and compares to chi-square with $d$ extra parameters.
One extra parameter at 5 percent means clearing 3.84.
Sixty of 100 heads gives 4.03 and rejects fair, matching the Wald value 2.0.
Wald and score are asymptotic siblings, and the ratio respects reparameterisation best.
Non-nested models need Vuong or held-out checks, not Wilks.
Boundary nulls halve the limit law, so correct the mixture or bootstrap.