Generalized Linear Models

Why Does This Exist?

Your helpdesk logs support tickets. Some days none, most days a handful, one memorable Tuesday forty-one. You have the promotional emails marketing sent that morning, and you want to know what each extra email costs you in tickets.

Reach for least squares and it hands you an answer. Ask what happens on a day with no promos and it predicts negative 4.8 tickets. Not rounded up to zero, not flagged. A number that cannot exist, stated with a straight face. And the negative is only half the trouble, because least squares also assumes constant spread, while quiet days vary by two or three tickets and busy days by fifteen.

Now look at what you already know. Linear regression predicts a mean directly and assumes bell-shaped noise. Logistic regression runs the same linear score through a sigmoid and assumes coin-flip noise. Those look like two models. They're one model with two dials set differently, and once you can see the dials you stop inventing a new loss every time your target changes shape. Both are fitted by maximum likelihood: pick the parameters that make the data you actually observed as unsurprising as possible.

Think of It Like This

How It Actually Works

Three parts, two of them choices

The first is the linear predictor, unchanged from least squares:

$\eta$ (eta) is a plain real number, one per row. $w$ holds the weights, $x$ one day's features, $b$ the intercept.

The second is the link function $g$, connecting $\eta$ to $\mu$, the expected value of the response for that row:

The inverse link $g^{-1}$ is what runs at prediction time, and it's where impossible values get ruled out. Pick $g = \log$ and its inverse is $e^{\eta}$, positive for every $\eta$ that ever existed. No negative ticket counts, structurally, not by clipping.

The third is the distribution family around $\mu$: Gaussian, Bernoulli, Poisson, Gamma. All belong to the exponential dispersion family, whose shape forces a mean-variance relationship out for free. Fix the family and you've fixed how the spread grows with the mean.

The three settings you'll actually use

Identity link, Gaussian family. $\mu = \eta$, constant variance, and maximising the likelihood reduces exactly to minimising squared error. That's ordinary least squares wearing a longer name.

Logit link, Bernoulli family. $\mu$ is a probability, $g(\mu) = \log\frac{\mu}{1-\mu}$, and the variance $\mu(1-\mu)$ peaks at 0.5 and vanishes at both ends. That's logistic regression.

Log link, Poisson family. $\mu = e^{\eta}$, and the variance equals the mean. This is the setting nobody teaches and everybody needs: tickets per day, clicks per hour, defects per batch. It answers the negative-4.8 problem and costs one keyword in most libraries.

When Poisson isn't enough

Poisson makes a testable promise: variance equals mean. Real count data usually breaks it. Days cluster, one outage spawns thirty tickets, an unmodelled feature shifts the whole rate, and the observed spread lands above the mean. That's overdispersion, and the damage is specific: coefficients stay roughly right while standard errors come out too small, so you report a significant effect that isn't there.

Check it on the residuals, not the raw column, since a raw variance-to-mean ratio of 10 mostly reflects the features doing their job. Conditioned on the fit, that ratio sits near 1 if Poisson holds. If it's 2, the honest fix is the negative binomial family, which is Poisson with a second parameter for exactly this slack.

Show Me the Code

Fit Poisson with Newton steps, then let least squares embarrass itself on the same 500 days.

1import numpy as np2
3def fit_poisson(X: np.ndarray, y: np.ndarray, steps: int = 50) -> np.ndarray:4    w = np.zeros(X.shape[1])5    for _ in range(steps):6        mu = np.exp(X @ w)  # the inverse log link, so every fitted mean stays positive7        w += np.linalg.solve(X.T @ (X * mu[:, None]), X.T @ (y - mu))8    return w9
10rng = np.random.default_rng(1)11emails = rng.uniform(0.0, 3.0, 500)                      # promo emails sent that day12X = np.column_stack([np.ones_like(emails), emails])13tickets = rng.poisson(np.exp(0.4 + 1.1 * emails))        # support tickets that day14w = fit_poisson(X, tickets)15ols = np.linalg.lstsq(X, tickets.astype(float), rcond=None)[0]16print(f"poisson: x{np.exp(w[1]):.2f} tickets per extra email, {np.exp(w[0]):.1f} at zero")17print(f"ols at zero emails: {ols[0]:.1f} tickets")18# -> poisson: x2.99 tickets per extra email, 1.5 at zero19# -> ols at zero emails: -4.8 tickets

Each extra email triples the ticket rate, and a quiet day sits at 1.5. The straight line predicts minus 4.8.

Watch Out For

The Quick Version

• Three parts: a linear predictor, a link tying it to the mean of the response, a distribution family for the noise.
• The link and the family are the two settings. Everything else is shared machinery, fitted by maximum likelihood.
• Identity + Gaussian is least squares. Logit + Bernoulli is logistic regression. Log + Poisson is count regression.
• Use a log link the moment your target can't be negative. Impossible predictions become unreachable.
• Overdispersion leaves coefficients right and standard errors too small. Negative binomial fixes it.
• Log-link coefficients multiply. $e^{w_j}$, never an additive shift.

What to Read Next

• Ordinary Least Squares is this framework with both dials at their defaults.
• Logistic Regression is the logit-plus-Bernoulli setting in full, including where its probabilities can be trusted.
• Maximum Likelihood Estimation is the engine underneath every setting, and why the link changes the loss.
• Loss Functions shows the same swap from the optimisation side, where Poisson deviance and Tweedie loss are these families in gradient-boosting clothing.
• Definitions worth a minute: Logarithm and Variance.