Lyapunov-based Safe RL

Lyapunov safe RL gives the agent an energy meter that must drop on every step, so only actions that move toward the goal count as safe.

Lyapunov rings shrink to the goal and only the move that lowers L from 4.0 to 2.25 passes while the move to 5.29 is blocked

Why Does This Exist?

Average safety lets one bad step hide inside a good average. A robot arm can snap a joint once and still pass a cost budget for the episode. A trading policy can breach risk for an hour and still look fine for the month.

Safe RL sets up that average view with constrained MDPs. This page adds the stricter per step certificate for when you need stability, not just a good mean. You will learn the decrease rule, the critic that enforces it, and the feasible set it protects. For broad planning with a model see model-based RL. For small policy steps without a certificate see TRPO.

The one idea you need first

A stage cost $c(x)$ is zero at the goal and positive elsewhere. A Lyapunov function $L(x)$ is an energy meter with $L(0) = 0$ and $L(x) > 0$ elsewhere. If you can force $L$ down each step, distance to goal shrinks and you cannot drift out.

Think of It Like This

A marble in a mixing bowl

Picture a marble in a smooth mixing bowl. Height is energy. The rim is the safety boundary and the bottom is the goal.

You allow only pushes that end lower than they start. A gentle push toward center passes because height falls from 4.0 to 2.25. A push up the wall fails because height rises from 4.0 to 5.29, even though the marble stays inside. Inside is not enough. It must settle lower.

Where it stops: a real bowl gives you gravity free. In RL you must learn the energy shape from data, and a bad estimate lets unsafe pushes slip through.

How It Actually Works

Call current state $x$ , action $a$ , next state $x'$ . Call $c(x)$ the stage cost, a non negative number that is zero only at goal. Call $\alpha$ a small positive margin rate.

The decrease condition

Certified stability needs this each step:

$L(x') - L(x) \leq -\alpha c(x)$

In words: next energy must sit below current energy by at least a margin. When the margin term is folded away, read it as $L(x') \lt L(x)$ . If an action breaks it, the filter blocks it and picks the closest passing action. No drop, no step.

The Lyapunov critic and the feasible set

In practice $L$ is a neural critic $L_\theta$ trained on $(x, a, x', c)$ to predict long term cost to goal. The policy picks reward seeking actions subject to:

$\mathbb{E}[L_\theta(x') \mid x, a] \leq L_\theta(x) - \alpha c(x)$

Here $\mathbb{E}$ averages over stochastic next states. The feasible set is one sublevel:

$\mathcal{S}_d = \{x \mid L(x) \leq d\}$

Pick $d$ so start sits inside. The rule makes the set invariant. Once inside, a certified policy can never climb out, because climbing needs $L$ to rise. That is stronger than a trust region, which limits step size but certifies no level.

Worked example with L(x) = x squared in 1D

Take $L(x) = x^2$ with goal $x = 0$ . Set $d = 9$ , so safe set is $|x| \leq 3$ . Start $x = 2.0$ :

$L(2.0) = 4.0$

Safe candidate $x' = 1.5$ :

$L(1.5) = 2.25$

$\Delta L = 2.25 - 4.0 = -1.75$

With $c(x) = 4.0$ and $\alpha = 0.2$ the required drop is $-0.8$ . Actual $-1.75$ clears it, so it passes.

Risky candidate $x' = 2.3$ :

$L(2.3) = 5.29$

$\Delta L = 5.29 - 4.0 = +1.29$

Positive, so blocked. Note the trap: $5.29 \leq 9$ , so it is still inside the set. Inside without a drop still fails, because repeated climbs would walk out. From $1.5$ to $1.0$ , $L$ falls $2.25$ to $1.0$ for $\Delta L = -1.25$ , again accepted. Repeated drops steer $L$ to zero.

Code

This is the exact filter check with the worked numbers.

1def lyapunov(x: float) -> float:2    """Energy L(x) = x squared."""3    return x * x4
5
6def passes_decrease(x: float, x_next: float, alpha: float = 0.2) -> bool:7    """True only when energy shrinks by the margin."""8    return lyapunov(x_next) <= lyapunov(x) - alpha * (x * x)9
10
11if __name__ == "__main__":12    print(f"L(start) = {lyapunov(2.0):.2f}")  # -> L(start) = 4.0013    print(f"L(safe) = {lyapunov(1.5):.2f}")  # -> L(safe) = 2.2514    print(f"safe passes = {passes_decrease(2.0, 1.5)}")  # -> safe passes = True15    print(f"L(risky) = {lyapunov(2.3):.2f}")  # -> L(risky) = 5.2916    print(f"risky passes = {passes_decrease(2.0, 2.3)}")  # -> risky passes = False

Watch Out For

An invalid candidate passes the code but proves nothing

A function with $L(0) \neq 0$ , negative values, or flat valleys away from goal can still shrink on your test batch. Symptom: training looks safe, then deployment parks at a non goal spot where $L$ bottoms out above zero.

Pin $L(0) = 0$ in code, use a positive head like a squared norm plus a small quadratic, and scan $L$ on a grid before trust. If any $x \neq 0$ gives $L(x) \leq 0$ , reject the candidate.

Checking slope instead of the next state drop

Learners copy $\dot{L} \lt 0$ from control books and check gradients in a discrete simulator. Symptom: slope points downhill at $x$ , yet the finite step lands uphill at $x'$ , so $L$ rises $4.0$ to $5.29$ while the slope check waved it through.

Always evaluate $L(x') - L(x)$ on the realized next state, averaged over stochastic outcomes. In discrete RL only the measured drop counts.

The Quick Version

Lyapunov safe RL certifies each step with energy $L$ that must shrink, so $L(x') \lt L(x)$ gates every action.
The feasible set is one energy level $\{x \mid L(x) \leq d\}$ , and the decrease rule keeps a certified policy inside it.
A critic $L_\theta$ learns long term cost to goal, and the policy may only pick actions with expected drop $\mathbb{E}[L(x')] \leq L(x) - \alpha c(x)$ .
In the 1D check with $L(x) = x^2$ , the move $2.0$ to $1.5$ drops $4.0$ to $2.25$ and passes, while $2.0$ to $2.3$ lifts to $5.29$ and is blocked.
Staying inside is not enough. An inside move still fails when it climbs uphill.