Transition Dynamics

Why Does This Exist?

Real environments ignore your intentions. A robot issues a drive command and hits slippery ice. A delivery drone banks right and meets a gust. A router forwards a packet and a buffer drops it. If your planner assumes each action lands exactly one next state, the first patch of noise shreds the plan.

Transition dynamics are the fix: for every state $s$ and action $a$, they give a probability distribution over next states $s'$. With that distribution you can compute expected returns, hedge against bad outcomes, and plan in uncertain worlds. Without it, the core machinery of Markov decision processes, Bellman equations, and value functions cannot even state what "good" means, because every one of them averages over what happens next.

Think of It Like This

How It Actually Works

The kernel: one distribution per state and action

Write $P(s' \mid s, a)$ for the probability that the environment lands in $s'$ at time $t+1$ given state $s$ and action $a$ at time $t$:

$P(s' \mid s, a) \doteq \Pr(S_{t+1} = s' \mid S_t = s, A_t = a)$

This rests on the Markov property: the distribution depends only on the current pair $(s, a)$, never on the longer history. That is what makes the object small enough to work with.

The textbook joint form $p(s', r \mid s, a)$ adds the reward $r$ into the same distribution: the probability of landing in $s'$ collecting $r$. You recover the transition kernel by summing the reward away, $P(s' \mid s, a) = \sum_r p(s', r \mid s, a)$, and the expected immediate reward by weighting rewards the same way, $R(s, a) = \sum_r r \sum_{s'} p(s', r \mid s, a)$. One object feeds both planning and evaluation.

Rows must sum to one

Because $P(\cdot \mid s, a)$ is a probability distribution, every row sums to exactly 1 and no entry goes negative:

$\sum_{s'} P(s' \mid s, a) = 1 \quad \text{for every } s, a$

For a finite state space with $n$ states, each action $a$ owns an $n \times n$ transition matrix $\mathbf{P}^a$ whose entry $(i, j)$ is $P(s_j \mid s_i, a)$. Every row sums to 1, which makes it a right-stochastic matrix. Follow a fixed policy $\pi$ and the whole system collapses to one matrix, $[\mathbf{P}^\pi]_{i,j} = \sum_a \pi(a \mid s_i) P(s_j \mid s_i, a)$, so a state distribution $\boldsymbol{\mu}_t$ evolves by plain multiplication, $\boldsymbol{\mu}_{t+1} = \boldsymbol{\mu}_t \mathbf{P}^\pi$. Probability mass is never created or destroyed, only moved.

Worked example: two steps down an icy corridor

Three states: $s_1$ Safe Road, $s_2$ Icy Patch, $s_3$ Goal (absorbing, so it keeps itself with probability 1). One action, Forward, with matrix:

$\mathbf{P}^{\text{Forward}} = \begin{bmatrix} 0.10 & 0.70 & 0.20 \\ 0.20 & 0.30 & 0.50 \\ 0.00 & 0.00 & 1.00 \end{bmatrix}$

Row check: $0.10 + 0.70 + 0.20 = 1.00$, $0.20 + 0.30 + 0.50 = 1.00$, $0 + 0 + 1 = 1.00$. Start certain in $s_1$, so $\boldsymbol{\mu}_0 = [1.00, 0.00, 0.00]$.

After one step, $\boldsymbol{\mu}_1 = \boldsymbol{\mu}_0 \mathbf{P} = [0.10, 0.70, 0.20]$, just the first row. After two steps, multiply again:

$\mu_{2,1} = 0.10(0.10) + 0.70(0.20) + 0.20(0.00) = 0.01 + 0.14 = 0.15$ $\mu_{2,2} = 0.10(0.70) + 0.70(0.30) + 0.20(0.00) = 0.07 + 0.21 = 0.28$ $\mu_{2,3} = 0.10(0.20) + 0.70(0.50) + 0.20(1.00) = 0.02 + 0.35 + 0.20 = 0.57$

So $\boldsymbol{\mu}_2 = [0.15, 0.28, 0.57]$, summing to 1.00, with a 57 percent chance of already holding the goal. These are the numbers in the diagram above.

Now use the same row to price an action. With values $V = [2.00, 5.00, 10.00]$, step reward $-1.00$, and discount $\gamma = 0.90$:

$\mathbb{E}[V(s')] = 0.10(2.00) + 0.70(5.00) + 0.20(10.00) = 0.20 + 3.50 + 2.00 = 5.70$ $Q(s_1, \text{Forward}) = -1.00 + 0.90 \times 5.70 = -1.00 + 5.13 = 4.13$

That weighting, probabilities times future values, is exactly what a Bellman update does. Without the dynamics row, the expectation cannot be written down.

Code

The snippet propagates the distribution two steps and prices the action, printing the same numbers the worked example derived.

1from typing import List2
3
4def step_distribution(mu: List[float], row_of: List[List[float]]) -> List[float]:5    """One distribution step: mu_{t+1}[j] = sum_i mu_t[i] * P[i][j]."""6    n = len(row_of)7    return [sum(mu[i] * row_of[i][j] for i in range(n)) for j in range(n)]8
9
10if __name__ == "__main__":11    P = [[0.10, 0.70, 0.20], [0.20, 0.30, 0.50], [0.00, 0.00, 1.00]]12    mu0 = [1.0, 0.0, 0.0]13    mu1 = step_distribution(mu0, P)14    mu2 = step_distribution(mu1, P)15    print([round(p, 2) for p in mu1])  # -> [0.1, 0.7, 0.2]16    print([round(p, 2) for p in mu2])  # -> [0.15, 0.28, 0.57]17    values = [2.0, 5.0, 10.0]18    expected_next = sum(p * v for p, v in zip(P[0], values))19    q = -1.0 + 0.9 * expected_next20    print(round(expected_next, 2))  # -> 5.721    print(round(q, 2))  # -> 4.13

Watch Out For

The Quick Version

• Transition dynamics $P(s' \mid s, a)$ assign every state-action pair a probability distribution over next states, which is the environment's physics in equation form.
• The Markov property keeps the object small: the next distribution depends only on the current state and action, not the history.
• Every row of every transition matrix sums to exactly 1, so state distributions evolve by multiplication with no mass created or lost.
• Bellman updates weight future values by these probabilities (here $0.10, 0.70, 0.20$ giving $Q = 4.13$); model-based RL plans with the matrix while model-free methods sample from it.
• Deterministic planning on stochastic hardware fails at the thinnest margin, and rows that do not sum to one corrupt every value built on them.

What to Read Next

• Markov Decision Processes
• Bellman Equations
• Value Functions