Deep Reinforcement Learning

Why Does This Exist?

Tabular methods need one entry per state. With an 84 by 84 grayscale camera frame the count of possible images is $256^{7056}$, far beyond any table. The agent also never sees the same exact frame twice, so memorization cannot generalize.

Linear approximation helped but moved the work to people. Engineers hand built tile codings and basis functions, and those features broke under lighting shifts or new viewpoints. Raw video, lidar and torque streams need features that adapt with the task.

Deep RL joins representation learning to control. One network maps raw input to compact features $z$ and then to values or a policy, trained end to end by reward. Early layers learn edges and motion, later layers learn decisions. This page frames that joint loop. For the value instance see deep Q-networks. For the policy instance see policy gradient. For the memory trick that stabilizes it see experience replay.

The one idea you need first

A Markov decision process has states $\mathcal{S}$, actions $\mathcal{A}$, transition odds $\mathcal{P}$, rewards $\mathcal{R}$ and discount $\gamma$. A parametrized function $f_{\theta}(s)$ with weights $\theta$ replaces the table. In deep RL that function is a neural network trained by reward feedback.

Think of It Like This

How It Actually Works

Write the network as an encoder $g_{\psi}$ plus a head $h_{\phi}$:

Here $s$ is the raw observation, $z = g_{\psi}(s)$ is the compact feature vector and $\theta$ collects $\psi$ and $\phi$. The head outputs either action values $Q_{\theta}(s, a)$ or a policy $\pi_{\theta}(a \mid s)$. Both train from the same reward stream that also shapes the encoder.

Three decision styles on shared features

• Value style learns $Q_{\theta}(s, a)$ against the Bellman target $y = r + \gamma \max_{a^{\prime}} Q_{\theta^{-}}(s^{\prime}, a^{\prime})$ from a frozen target net.
• Policy style climbs expected return $J(\theta)$ with the score gradient $\nabla_{\theta} \log \pi_{\theta}(a \mid s)$ times return or advantage.
• Actor critic style keeps both: the actor proposes $\pi_{\theta}$ and the critic $V_{\phi}$ judges with bootstrapped targets to cut variance.

All three reuse the same encoder, which is why perception and control improve together.

Why training is harder than supervised learning

Targets move because $y$ depends on the same weights being trained. Data correlates because $s_{t+1}$ follows $s_t$. Representations shift because new rooms overwrite old features. The trio of function approximation plus bootstrapping plus off policy data can diverge, a pattern called the deadly triad. Replay buffers, frozen targets and clipped objectives exist to tame exactly these effects.

Worked example

Take $s = [1.0, 0.5]$ with two actions, discount $\gamma = 0.90$ and step size $\alpha = 0.1$. Hidden width $2$ with ReLU. Weights:

Forward pass. Pre activations:

ReLU keeps both, so $h = [0.50, 0.60]$. Outputs:

Transition. Act $a = 0$, earn $r = 1.0$. Target net rates the next state best at $0.80$. Target and error:

One gradient step with $\alpha = 0.1$ moves $Q(s, 0)$ from $0.30$ toward $1.72$. The same backward pass also reshapes the encoder that built $h$, so perception and values learn from one number.

Code

This snippet repeats the forward pass and Bellman numbers with plain Python.

1def relu(v: float) -> float:2    """Rectified linear unit."""3    return v if v > 0.0 else 0.04
5
6s = [1.0, 0.5]7W1 = [[0.5, -0.2], [0.4, 0.6]]8b1 = [0.1, -0.1]9W2 = [[0.8, -0.5], [0.3, 0.7]]10b2 = [0.2, -0.3]11
12z1 = [W1[0][0] * s[0] + W1[0][1] * s[1] + b1[0], W1[1][0] * s[0] + W1[1][1] * s[1] + b1[1]]13print([round(v, 4) for v in z1])  # -> [0.5, 0.6]14h = [relu(v) for v in z1]15print([round(v, 4) for v in h])  # -> [0.5, 0.6]16Q = [W2[0][0] * h[0] + W2[0][1] * h[1] + b2[0], W2[1][0] * h[0] + W2[1][1] * h[1] + b2[1]]17print([round(v, 4) for v in Q])  # -> [0.3, 0.27]18y = 1.0 + 0.90 * 0.8019print(f"{y:.4f}")  # -> 1.720020td = y - Q[0]21print(f"{td:.4f}")  # -> 1.420022print(f"{0.5 * td * td:.4f}")  # -> 1.0082

Hidden values, Q values, target, error and loss match the hand pass, which ties the diagram to the prose.

Watch Out For

The Quick Version

• Deep RL trains $f_{\theta}(s) = h_{\phi}(g_{\psi}(s))$ end to end, so features $z$ and decisions improve from one reward signal.
• Value, policy and actor critic heads share the encoder and differ only in what the head predicts.
• Moving Bellman targets, correlated streams and shifting features make training less stable than supervised fits.
• Worked numbers: $s = [1.0, 0.5]$, $h = [0.50, 0.60]$, $Q = [0.30, 0.27]$, $y = 1.72$, loss $1.0082$.
• Replay, frozen targets and clipped updates are the standard stabilizers, not optional extras.
• Hand features fail on raw high dimensional input, which is why the encoder earns its keep.

What to Read Next

• Deep Q-Networks
• Experience Replay
• Policy Gradient Methods
• Recurrent Policies with Memory in RL