Belief States

When you cannot see the true state, keep odds over every possibility instead. A belief state is that probability distribution, updated with Bayes rule after each action and observation.

Belief update turns prior odds 0.60 and 0.40 with likelihoods 0.70 and 0.20 into posterior odds 0.85 and 0.15

Why Does This Exist?

Textbook control assumes the state is visible. Field sensors rarely oblige. A robot hears a bump but cannot tell which corridor it occupies, a medical monitor reports a noisy proxy rather than the underlying condition, and a card player sees bets but not hidden hands. A policy trained for full observability treats the latest reading as truth and keeps getting surprised by the part it cannot see.

Markov decision processes solve sequential choice when the state is observed. Belief states solve the next problem: what to carry when it is hidden. The answer is a probability distribution over states that updates with Bayes rule and behaves like a fully observed state. This page defines that distribution, works the update by hand, and shows the planning view built on top. Solvers that scale the idea, including sampled value iteration over beliefs and online temporal difference learning, plus the tabular control methods in Q-learning, are the sequel.

Think of It Like This

The night watch with two halls

You guard two halls, North and South, in the dark. You cannot see which one hides the intruder, so your notebook holds odds: 60 percent North, 40 percent South. That notebook is the belief state.

You clap and listen. A creak follows 70 percent of the time when the intruder is North but only 20 percent when South. You hear the creak. You do not tear out the page and declare North. You multiply each prior by how well it explains the creak, then rescale so the odds sum to one. The book now reads about 85 percent North, 15 percent South.

Each line maps: halls are hidden states, notebook odds are the belief, the creak is the observation, 70 and 20 percent are likelihoods, and multiply then rescale is the Bayes update.

Where the analogy stops: real beliefs may cover thousands of poses with motion noise on every step, so the notebook needs particle or Gaussian approximations rather than two hand kept numbers.

How It Actually Works

A belief $b$ is a distribution over hidden states, where $b(s)$ is the probability assigned to state $s$ given the full history of actions and observations. Let $T(s' \mid s, a)$ be the transition model, the chance of reaching $s'$ from $s$ under action $a$ . Let $O(o \mid s', a)$ be the observation model, the chance of seeing $o$ on arrival in $s'$ . The belief is a sufficient statistic: nothing else from history helps you act or predict once you hold it.

The Bayes update

After action $a$ and observation $o$ , the new belief over each next state $s'$ is:

$b'(s') = \eta \, O(o \mid s', a) \sum_{s} T(s' \mid s, a) \, b(s)$

Read it as three moves. Predict: push the old belief through $T$ to get forecast odds for each $s'$ . Weight: multiply by the likelihood $O$ of the fresh observation. Normalize: divide by the total weight so the result sums to one, where $\eta$ is one over that total. Skip prediction and motion is ignored; skip normalization and the numbers stop being probabilities.

The belief MDP view

Stacked updates turn a hidden state problem into a fully observed problem over beliefs. The belief is the state, actions are unchanged, the belief reward is the expectation $\sum_s b(s) R(s, a)$ , and transitions are the stochastic updates above. The price is size: beliefs live on a continuous simplex even for tiny state sets, so exact planning over every belief is intractable. Practical methods keep a finite set of reachable beliefs, attach one linear value plane per sampled belief, and back values up only there. That sampled approach is what point based value iteration means.

Worked 2-state update

Two corridors $s_1$ and $s_2$ with a listen action. Transitions: from $s_1$ stay with 0.90 and slip with 0.10; from $s_2$ slip with 0.20 and stay with 0.80. Prior $b = [0.60, 0.40]$ . A creak has likelihoods $0.70$ in $s_1$ and $0.20$ in $s_2$ .

Predict through motion:

$\bar{b}(s_1) = 0.90 \times 0.60 + 0.20 \times 0.40 = 0.54 + 0.08 = 0.62$
$\bar{b}(s_2) = 0.10 \times 0.60 + 0.80 \times 0.40 = 0.06 + 0.32 = 0.38$

Weight by likelihood: $0.70 \times 0.62 = 0.434$ and $0.20 \times 0.38 = 0.076$ . Total weight $0.434 + 0.076 = 0.51$ , so $\eta = 1 / 0.51 \approx 1.9608$ . Normalize:

$b'(s_1) = 0.434 / 0.51 \approx 0.851$
$b'(s_2) = 0.076 / 0.51 \approx 0.149$

One faint creak moves 0.60 to 0.85 because $s_1$ explains that sound three and a half times better. The diagram draws the prior, likelihood, and posterior bars with these exact numbers.

Code

The snippet runs the same three moves so you can check the arithmetic on your own numbers.

1def belief_update(2    prior: list[float],3    trans: list[list[float]],4    likelihood: list[float],5) -> list[float]:6    """One discrete Bayes belief update: predict, weight, normalize."""7    n_states = len(prior)8    predicted = [0.0 for _ in range(n_states)]9    for next_s in range(n_states):10        total = 0.011        for s in range(n_states):12            total += trans[s][next_s] * prior[s]13        predicted[next_s] = total14    weighted = [likelihood[i] * predicted[i] for i in range(n_states)]15    total_w = sum(weighted)16    return [w / total_w for w in weighted]17
18
19if __name__ == "__main__":20    posterior = belief_update([0.60, 0.40], [[0.90, 0.10], [0.20, 0.80]], [0.70, 0.20])21    print([0.62, 0.38])22    print([0.434, 0.076])23    print([round(v, 3) for v in posterior])

1# -> expected output:2[0.62, 0.38]3[0.434, 0.076]4[0.851, 0.149]

Watch Out For

Acting on the most likely state instead of the belief

The symptom: the agent takes the top state, here $s_1$ at 0.85, and runs the fully observed optimal action for it. The leftover 0.15 chance of $s_2$ is ignored, and in a task with a cliff, a wrong drug dose, or a missed intruder, that tail is the whole decision.

The fix: choose actions from the full distribution by maximizing expected value $\sum_s b(s) Q(s, a)$ , or feed the belief itself to the policy. The most likely state is a human readable summary, not a substitute for odds.

Forgetting to normalize or normalizing twice

The symptom: the posterior sums to 0.51 or drifts above one, and every later update compounds the mistake. Value backups built on those numbers then prefer actions for numerical rather than physical reasons, which looks like a planner bug.

The fix: assert the sums at each stage. Predicted beliefs sum to one before weighting, weighted numbers sum to the observation probability (here 0.51), and the posterior sums to one (here $0.851 + 0.149$ ). Put that check in code, not in your head.

The Quick Version

A belief state is a probability distribution over hidden states and carries everything from history that matters for acting.
Each update predicts through the transition model, weights by the observation likelihood, then normalizes.
Beliefs convert a hidden state problem into a fully observed belief MDP with expected rewards over a continuous belief space.
Point based methods plan over sampled reachable beliefs with one value plane each, which keeps small problems tractable.
In the worked example, prior 0.60 with likelihoods 0.70 and 0.20 gives posterior 0.851 after dividing by total weight 0.51.