Radial Basis Functions in RL

Radial basis functions tile a continuous state space with smooth overlapping bells instead of sharp boxes, so the value surface stays differentiable and a small step in any direction gives a small sensible change.

Three Gaussian bumps at 0.0, 0.5 and 1.0 turn state 0.60 into features 0.14, 0.95 and 0.41, which weights 2, 5 and 8 turn into value 8.29

Why Does This Exist?

Robot joints, velocities, and torques vary continuously, but the classic linear features are binary: a state is either inside a tile or outside it. Cross a tile coding boundary and the value estimate jumps like a staircase, flat plates separated by cliffs. On the plates the spatial gradient is exactly zero, so an actor-critic gets no direction to follow; at the edges the gradient does not exist at all.

Radial basis functions fix the representation, not the algorithm. Each feature is a smooth bell, usually Gaussian, centered at a prototype state. A state near the center activates the feature strongly, a far state barely at all, and everything in between varies continuously. The value surface becomes infinitely differentiable, the spatial gradient exists everywhere, and policy methods can climb it. Sutton and Barto present RBFs as the continuous generalization of coarse coding, and the formula below is theirs.

This page covers the Gaussian form, the update, the bandwidth trade-off, and one fully traced state. For the binary tiling it replaces, see tile coding. For the linear methods both plug into, see value function approximation.

Think of It Like This

Stage lights with soft lenses

Three spotlights hang over a dark stage, aimed at the left mark, center mark, and right mark. Each carries a soft diffusion lens instead of a sharp shutter: full brightness at its own mark, fading smoothly outward in a bell curve.

Stand just right of center. The center beam hits you at 95 percent, the right beam's halo reaches you at 41 percent, and the far left beam barely brushes you at 14 percent. Your total light is the blend of all three, and every step you take changes it gradually. Nothing flickers, nothing jumps, because there are no edges to cross.

Map it point for point: marks are basis centers, brightness percentages are feature activations, the blend is the weighted sum with learned weights, and the absence of shutters is differentiability.

Where it stops: stage light only adds, while learned weights can be negative. Dips and penalty valleys in a value surface need those negative weights, which have no lighting equivalent.

How It Actually Works

A continuous state $s$ becomes a feature vector $\phi(s) = [\phi_1(s), \dots, \phi_d(s)]$ through one Gaussian per center $c_i$ with width $\sigma_i$ :

\phi_i(s) = \exp\left(-\frac{\lVert s - c_i \rVert^2}{2\sigma_i^2}\right)

In words: exponentiate the negative squared distance, scaled by the width. At the center the activation is $1.0$ ; far away it decays toward zero. The value estimate is a plain weighted sum, $\hat{v}(s, w) = \sum_i w_i \phi_i(s)$ , and because the Gaussian is smooth, the spatial gradient exists in closed form:

\nabla_s \hat{v}(s, w) = \sum_i w_i \, \phi_i(s) \left(-\frac{s - c_i}{\sigma_i^2}\right)

Learning is the semi-gradient update: observe a target $U$ for the visited state, form the error $\delta = U - \hat{v}(s, w)$ , and move each weight in proportion to its activation, $w_i \leftarrow w_i + \alpha \, \delta \, \phi_i(s)$ . The nearest center absorbs most of the update, neighbors get a smooth share, and nobody gets a cliff.

Worked example: state 0.60 under three bells

Centers at $c = [0.0, 0.5, 1.0]$ with shared width $\sigma = 0.3$ , so $2\sigma^2 = 0.18$ . Weights start at $w = [2.0, 5.0, 8.0]$ , step size $\alpha = 0.1$ , observed target $U = 10.0$ .

First the activations at $s = 0.60$ :

\phi_1 = \exp(-0.36 / 0.18) = \exp(-2.0000) = 0.1353

\phi_2 = \exp(-0.01 / 0.18) = \exp(-0.0556) = 0.9460

\phi_3 = \exp(-0.16 / 0.18) = \exp(-0.8889) = 0.4111

Then the value, term by term:

\hat{v} = 2.0 \times 0.1353 + 5.0 \times 0.9460 + 8.0 \times 0.4111 = 0.2707 + 4.7298 + 3.2889 = 8.2894

The error is $\delta = 10.0 - 8.2894 = 1.7106$ , and each weight moves by $0.1 \times 1.7106 \times \phi_i$ :

w \leftarrow [2.0232, \; 5.1618, \; 8.0703]

The nearest center takes the largest step ( $+0.1618$ ) while the far one barely moves ( $+0.0232$ ), with no boundary deciding who learns. The diagram above draws these same three bells and the $8.2894$ they produce.

The one knob: bandwidth

Width $\sigma$ sets how far each bell reaches, and both extremes hurt. Too wide and every bell sees everything: sharp reward peaks wash out into a bland average. Too narrow and bells stop overlapping: states between centers activate nothing, predictions collapse toward zero, and gradients vanish in the dead zones. The working rule is to space centers evenly and set $\sigma$ near the center spacing, so neighbors cross around $0.5$ activation. Against binary tilings, the summary is short: RBFs cost a transcendental exponential per basis per step and go dense everywhere, while tile coding costs a few integer lookups and stays sparse, which is why tilings survive in high dimensions and RBFs own the low-dimensional smooth jobs.

Code

Plain standard library, matching the hand trace exactly:

1import math2
3
4def rbf_features(state: float, centers: list[float], sigma: float) -> list[float]:5    denom = 2.0 * sigma * sigma6    return [math.exp(-((state - c) ** 2) / denom) for c in centers]7
8
9centers = [0.0, 0.5, 1.0]10weights = [2.0, 5.0, 8.0]11state, target, alpha = 0.60, 10.0, 0.112
13phi = rbf_features(state, centers, 0.3)14value = sum(w * p for w, p in zip(weights, phi))15delta = target - value16weights = [w + alpha * delta * p for w, p in zip(weights, phi)]17
18print([round(p, 4) for p in phi])  # -> [0.1353, 0.946, 0.4111]19print(round(value, 4))             # -> 8.289420print(round(delta, 4))             # -> 1.710621print([round(w, 4) for w in weights])  # -> [2.0232, 5.1618, 8.0703]

Four prints, four matches with the worked example: features, value, error, and updated weights.

Watch Out For

Gridding centers in high dimensions

Even spacing sounds principled until you count. Five centers per dimension across an 8-dimensional arm needs $5^8 = 390{,}625$ bells, every one evaluated densely on every step, each costing distance math plus an exponential. Tile coding answers the same space with a handful of active tiles. The fix is to stop gridding: place centers on states the agent actually visits, cluster replay observations and plant bells there, and switch to tilings or neural features once dimensions outgrow a dozen.

Dead zones between narrow bells

When centers sit far apart relative to $\sigma$ , the gaps between them activate nothing: every $\phi_i$ rounds to zero, the value reads as zero, and the update $w_i + \alpha \delta \phi_i$ moves nothing, so the zone never learns its way out. The symptom is a value surface with confident predictions near centers and flat zero deserts between them. The fix is normalized activations that always sum to one, $\tilde{\phi}_i = \phi_i / \sum_j \phi_j$ , which guarantees every state belongs somewhere.

The Quick Version

Gaussian bells turn a continuous state into smooth features in $(0, 1]$ , replacing sharp tile memberships.
State $0.60$ under centers $[0.0, 0.5, 1.0]$ gives features $[0.1353, 0.9460, 0.4111]$ and value $8.2894$ under weights $[2.0, 5.0, 8.0]$ .
Target $10.0$ gives error $1.7106$ and new weights $[2.0232, 5.1618, 8.0703]$ , shared smoothly by distance.
The value surface is differentiable everywhere, so actor methods get usable spatial gradients.
Set width near center spacing; normalize activations or cluster centers to avoid dead zones and dimensional blowup.