Option-Critic Architecture

Why Does This Exist?

Long tasks need skills, not just twitches. Nobody solves a dungeon by choosing muscle torques; they cross rooms, grab keys, open doors. The options framework wrote that idea down in 1999: an option has an initiation set, an internal policy $\pi_\omega$ for acting while it runs, and a termination chance $\beta_\omega$ for handing control back.

The framework left one question bleeding: where do options come from? For years the answer was a human. People hand-placed bottleneck states, drew initiation sets around doorways, and tuned termination by feel. Two-phase pipelines clustered state graphs offline and froze the skills before learning began. Nothing adapted when the world changed, and no gradient connected final reward back to the quitting decisions.

Option-Critic closed the loop in 2017. Bacon, Harb, and Precup derived gradients for both the internal policies and the termination functions, so skills and quitting rules learn end to end from reward. For the framework it builds on, see options framework. For the broader family, see hierarchical RL. For the value-decomposition sibling, see MAXQ.

The one idea you need first

An option's value $Q_\Omega(s, \omega)$ is the expected return of starting option $\omega$ in state $s$. The state's hierarchical value $V_\Omega(s)$ is the best the agent can do from $s$ by picking options well. Their gap, the advantage $A_\Omega(s, \omega) = Q_\Omega(s, \omega) - V_\Omega(s)$, says whether staying in the current option beats quitting. Negative means quit, positive means persist. Everything below is gradient ascent listening to that gap.

Think of It Like This

How It Actually Works

One shared trunk reads the state; three heads decide. The intra-option head $\pi_{\omega}(a \mid s)$ picks primitive actions while option $\omega$ runs. The termination head $\beta_\omega(s)$ outputs a quit chance, usually a sigmoid over a logit. The critic head $Q_\Omega(s, \omega)$ scores options.

Arriving in $s'$ under option $\omega$, the continuation value blends staying and quitting:

The intra-option policy climbs the gradient of $Q_U(s, \omega, a) = r(s, a) + \gamma U(s', \omega)$, favoring actions that lead somewhere the current option stays valuable. The termination parameters climb a different gradient, the termination gradient theorem:

Read the sign: a negative advantage (current option losing) flips the minus and pushes $\beta$ up, so the option quits. A positive advantage pushes $\beta$ down, so the skill persists. Raw switching is free in this math, so unregularized agents quit everywhere and skills collapse to single steps. A deliberation cost $\eta > 0$ added to the advantage demands the gain beat a switching fee before quitting pays.

Worked example

State $s'$ with active option $\omega_1$, rival $\omega_2$, deliberation cost $\eta = 0.5$, step size $\alpha = 0.5$:

• Option values: $Q(s', \omega_1) = 4.00$, $Q(s', \omega_2) = 6.00$, so $V(s') = 6.00$.
• Raw advantage: $4.00 - 6.00 = -2.00$. Regularized: $-2.00 + 0.50 = -1.50$. Staying loses even after the fee.
• Termination logit $z = 0$, so $\beta = \sigma(0) = 0.50$ with slope $\beta(1 - \beta) = 0.25$.
• Ascent direction: $-0.25 \times (-1.50) = +0.375$. Update: $\Delta z = 0.5 \times 0.375 = +0.1875$, so $z = 0.1875$.
• New quit chance: $\beta = \sigma(0.1875) = 1 / (1 + e^{-0.1875}) \approx 0.5467$, up from 0.50.

The losing option now quits more often, exactly as the advantage ordered.

Code

This replay runs the worked numbers above:

1import math2
3
4def sigmoid(z: float) -> float:5    """Logistic squashing from logit to probability."""6    return 1.0 / (1.0 + math.exp(-z))7
8
9def termination_update(q_current: float, q_best: float, eta: float, z: float, alpha: float) -> tuple[float, float, float]:10    """Advantage, gradient step, and new termination probability."""11    advantage = (q_current - q_best) + eta12    beta = sigmoid(z)13    grad = beta * (1.0 - beta) * advantage14    z_new = z - alpha * grad15    return round(advantage, 2), round(z_new, 4), round(sigmoid(z_new), 4)16
17
18print(termination_update(4.0, 6.0, 0.5, 0.0, 0.5))  # -> (-1.5, 0.1875, 0.5467)

The output matches the hand walk: advantage $-1.50$, logit $0.1875$, quit chance $0.5467$.

Watch Out For

The Quick Version

• Option-Critic learns intra-option policies, termination chances, and option values together from reward gradients, with no hand-built subgoals.
• The intra-option policy climbs $Q_U$ built on the continuation value $U$, which blends staying with quitting through $\beta$.
• The termination gradient pushes $\beta$ up when the advantage is negative and down when it is positive; here $-1.50$ moves 0.50 to 0.5467.
• Free switching collapses options to single steps, so a deliberation cost $\eta$ charges a fee for quitting.
• Bootstrap intra-option updates through $U$, never through plain action values, or the gradient credits phantom value.

What to Read Next

• The Options Framework
• Hierarchical RL
• MAXQ Value Function Decomposition