Kernel Density Estimation

Histograms jump with bin edges. KDE centers a smooth bump on every point and averages them into a curve with no bins to argue about.

Eight samples each add a bump of width 0.3, averaging to 0.34 at x equals 2.1 with no bin edges

Why Does This Exist?

Histograms lie by binning: shift the edges and modes appear or vanish, while high dimensions make bins explode exponentially. Parametric fits lie by shape, because data are rarely exactly Gaussian. Analysts need a smooth, bin-free picture of where data concentrate before modeling.

KDE delivers it with $\hat{f}_h(x) = \frac{1}{n h} \sum_i K((x - X_i) / h)$ : one kernel $K$ per point, averaged, with bandwidth $h$ setting bump width and Gaussian kernels as the default. It needs only descriptive statistics as background and pairs with resampling in bootstrap and resampling.

The one idea you need first

A density integrates to 1 and runs tall where data crowd. Each kernel bump carries $1/n$ mass, so the average integrates to 1 automatically. Bandwidth is the only knob: small $h$ follows every point, large $h$ flattens everything.

Think of It Like This

Sand piles instead of buckets

A histogram sorts sand into buckets, so moving the walls changes the skyline. KDE drops one small sand pile on each grain and lets them merge into dunes.

Pile width is bandwidth: fine piles spike at every grain, wide piles flatten the desert. The mapping holds pile for kernel and dune for density.

Where it stops: sand piles are physical and local, while KDE sums globally per query, which turns expensive on large $n$ without trees or FFT.

How It Actually Works

Bias shrinks as $h^2$ while variance shrinks as $1 / (n h)$ , the classic tradeoff minimized near $h \sim n^{-1/5}$ in one dimension with integrated squared error $\sim n^{-4/5}$ . Silverman rule $h = 0.9 \min(\sigma, \mathrm{IQR} / 1.34) n^{-1/5}$ starts Gaussian-ish cases. Multivariate KDE needs full bandwidth matrices and degrades fast with dimension: effective samples shrink exponentially, so past about five dimensions prefer parametric or projection methods.

Worked example

Eight points $[1.0, 1.5, 2.0, 2.2, 3.0, 3.5, 5.0, 5.2]$ with Gaussian $h = 0.3$ at $x = 2.1$ . The normalizer is $8 \times 0.3 \times \sqrt{2 \pi} \approx 6.016$ .

Neighbors dominate: for $2.0$ and $2.2$ , $u = 0.1 / 0.3 \approx 0.333$ and $\exp(-0.5 u^2) \approx 0.946$ each. Each contributes $0.946 / 6.016 \approx 0.157$ . Next points add little: $1.5$ gives $u = 2.0$ with $\exp(-2) \approx 0.135$ for $0.022$ , $3.0$ gives $u = -3.0$ with $\exp(-4.5) \approx 0.011$ for $0.002$ , and $5.0$ contributes essentially zero. Summing all eight gives $\hat{f}(2.1) \approx 0.34$ .

Bandwidth moves the answer as theory predicts. With $h = 0.05$ the same $x$ falls to about $0.27$ while points spike near $1.0$ . With $h = 2$ the curve flattens near $0.12$ to $0.15$ everywhere.

Code

The snippet evaluates the same Gaussian KDE at one grid point, so the hand total can be checked directly.

1import numpy as np2
3x = np.array([1.0, 1.5, 2.0, 2.2, 3.0, 3.5, 5.0, 5.2])4h = 0.35grid = np.array([2.1])6dens = np.exp(-0.5 * ((grid[:, None] - x) / h) ** 2).sum(axis=1)7dens = dens / (x.size * h * np.sqrt(2 * np.pi))8print(round(float(dens[0]), 2))  # -> 0.34

Hand sum and code agree at 0.34. Change $h$ to 0.05 or 2.0 and watch spike versus flatten.

Watch Out For

Trusting default bandwidths blindly

Silverman assumes near-Gaussian data and oversmooths multimodality: two real humps merge into one and analysts conclude one segment. Undersmoothed defaults do the reverse and invent modes from noise.

Plot several bandwidths across a decade of $h$ . When the mode count changes, the default decided the conclusion, not the data.

KDE on high dimensions or bounded edges

In ten dimensions neighborhoods sit empty and bumps never overlap, so the curve is $n$ spikes on a plain. On bounded $[0, \infty)$ data, symmetric kernels leak mass across the edge and understate near-zero density.

Cap KDE at low dimensions, use boundary correction or logs near edges, and switch families past about five dimensions. When most queries see one bump, dimensionality won.

The Quick Version

KDE averages one $1/n$ -mass bump per point, with no bins and one bandwidth knob.
Bias scales as $h^2$ and variance as $1 / (n h)$ , with optimal $h \sim n^{-1/5}$ in one dimension.
The eight-point example at $h = 0.3$ gives $\hat{f}(2.1) \approx 0.34$ .
Tiny $h$ spikes on noise while huge $h$ flattens real modes, so scan a decade of bandwidths.
Past about five dimensions or on hard boundaries, correct the edges or switch methods.
Naive queries cost $O(n)$ each, so use trees or FFT at scale.