Adversarial Attacks on RL Policies

A sticker-sized change to what the agent sees can flip what it does. Tiny crafted observation shifts push a trained policy into the wrong action.

A clean observation scores logits 0.10 and minus 0.10 for action 0, while an epsilon 0.06 shift flips them to minus 0.02 and 0.02 for action 1

Why Does This Exist?

Game agents that read raw pixels look unbeatable until a few pixels change. In 2017 Huang and coauthors showed the gradient tricks that fool image classifiers also fool trained game policies: an invisible shift can tank a Deep Q-Networks score or steer it into its worst move. Off the bench that failure mode follows the policy out the door. A robot reading a doctored sign or a trading agent fed shifted prices breaks the same way.

This page is about attackers, not accidents. Safe RL covers respecting constraints when nobody is out to get you. Here you learn deliberate pushes: perturbed observations at test time, strikes timed for the moments that count, and poisoned signals during training.

One prerequisite first. A value-based agent scores every action with Q-values and picks the largest. When pixels drive those scores, pixels can rig the vote. The full treatment lives in deep reinforcement learning.

Think of It Like This

The doctored stop sign

Researchers once stuck small patches on a stop sign. People still read STOP without thinking, while a vision model read a speed limit sign and acted on it. The sign is the observation $s$ , the patches are the perturbation $\delta$ inside a budget $\epsilon$ , the driver is the policy $\pi$ , and the missed stop is the flipped action.

That maps point for point onto a test-time observation attack. The attacker never touches the weights. They edit only the input, stay inside a budget a person would ignore, and let the trusting policy do the rest.

Where it stops: a sticker is visible and one sign cannot fight back across time. In RL the attacker can also strike at just two or three decisive steps of an episode, or poison training itself, and each wrong action compounds into the next state.

How It Actually Works

Symbols for this page: $s$ is the observed state such as game pixels, $a$ the action, $\pi(a \mid s)$ the policy, $Q(s, a)$ the action scores, $L$ the attacker loss, $\delta$ the added perturbation, and $\epsilon$ the per-pixel budget.

Observation perturbations and FGSM

The fast gradient sign method builds $\delta$ in one step:

s_{adv} = s + \epsilon \cdot \mathrm{sign}(\nabla_s L)

Read it inside out. $\nabla_s L$ says which way each pixel should move to help the attacker. The sign keeps only the direction per pixel. Budget $\epsilon$ sets the stride. Every pixel moves at most $\epsilon$ , so the change stays near invisible while the logits shift enough to flip the argmax. Classifiers push away from the true label. Policy attackers push away from the currently preferred action instead, for instance by widening the gap between a target action score and the best score.

Worked example with epsilon 0.06 flipping the action

Two actions share linear weights $w_0 = [1, -1]$ and $w_1 = [-1, 1]$ . Logits are $z_0 = w_0 \cdot s$ and $z_1 = w_1 \cdot s$ , and the agent picks the larger.

Clean observation $s = [0.60, 0.50]$ :

z_0 = 0.60 - 0.50 = 0.10, \quad z_1 = -0.60 + 0.50 = -0.10

Softmax exponentials are $[1.1052, 0.9048]$ summing to $2.01$ , so probabilities are $[0.5498, 0.4502]$ . Action 0 wins near 55 percent.

The attacker wants action 1 with loss $L = z_1 - z_0$ . Its gradient is $w_1 - w_0 = [-2, 2]$ with sign $[-1, +1]$ . At budget $\epsilon = 0.06$ :

\delta = 0.06 \cdot [-1, +1] = [-0.06, +0.06], \quad s_{adv} = [0.54, 0.56]

Fresh logits:

z_0 = 0.54 - 0.56 = -0.02, \quad z_1 = -0.54 + 0.56 = 0.02

Exponentials $[0.9802, 1.0202]$ sum to $2.0004$ , so probabilities are $[0.49, 0.51]$ . Action 1 wins. Each input moved only $0.06$ yet the choice flipped.

The margin explains it. The clean gap was $0.10 - (-0.10) = 0.20$ . The attack moved the gap by $(w_1 - w_0) \cdot \delta = (-2)(-0.06) + (2)(0.06) = 0.24$ , which beats $0.20$ . Small push, right direction, closed gap.

Striking only when it counts

Perturbing every frame is noisy and easy to spot, so timed attackers hit only the steps where the preference is strong but brittle and leave the rest clean. They score each step by how much the outcome hinges on it, spend the budget there, and collect most of the damage for a fraction of tampered frames. Defenses that scan for constant noise miss this, because nearly the whole episode looks untouched.

Poisoning training instead of testing

Test-time attacks edit what the agent sees. Training-time attacks edit what it learns from. Flipped rewards, relabeled good actions, or a trigger pattern hidden in some training states plant a backdoor: normal behavior until the trigger appears, then misbehavior on cue. Locked datasets, reward sanity checks, and demonstration provenance are the counter. The standard hardening is adversarial training, perturbing observations during training and verifying against a held-out attacker the training never saw.

Code

A short check of the worked flip above, with type hints throughout:

1import math2from typing import List3
4
5def softmax(z: List[float]) -> List[float]:6    exps = [math.exp(v) for v in z]7    total = sum(exps)8    return [e / total for e in exps]9
10
11def logits(state: List[float]) -> List[float]:12    w0 = [1.0, -1.0]13    w1 = [-1.0, 1.0]14    z0 = w0[0] * state[0] + w0[1] * state[1]15    z1 = w1[0] * state[0] + w1[1] * state[1]16    return [z0, z1]17
18
19s = [0.60, 0.50]20eps = 0.0621
22clean_p = softmax(logits(s))23print([round(p, 4) for p in clean_p])24# -> [0.5498, 0.4502]25
26grad_sign = [-1.0, 1.0]27s_adv = [s[i] + eps * grad_sign[i] for i in range(2)]28print([round(v, 2) for v in s_adv])29# -> [0.54, 0.56]30
31adv_p = softmax(logits(s_adv))32print([round(p, 4) for p in adv_p])33# -> [0.49, 0.51]

1# -> expected output:2[0.5498, 0.4502]3[0.54, 0.56]4[0.49, 0.51]

Watch Out For

Grading the agent in a clean room only

A policy can score perfectly on the untouched environment and still fold under tiny deployment noise. The failure lands on the close calls you never tested, which are usually the highest stakes states.

Test under fire before trusting a score. Report FGSM and multi-step attacker results at a stated epsilon next to the clean number, plus plain corruptions like blur and quantization.

Gradient masking that feels like safety

Hardening can flatten the attacker gradient so their FGSM fails, and teams declare victory. Then a black-box attacker who never needed those gradients, or the same attack retrained against the new weights, walks through. Hidden gradients are obscurity, not safety.

Verify with attackers that skip your gradients: surrogate transfer, score queries, and random search inside the epsilon ball. Pair that with adversarial training instead of gradient shaping.

The Quick Version

Tiny input shifts flip RL actions: FGSM adds $\epsilon \cdot \mathrm{sign}(\nabla_s L)$ and the argmax changes.
One checkable flip: clean logits $[0.10, -0.10]$ pick action 0 near 55 percent, and $\epsilon = 0.06$ gives $[-0.02, 0.02]$ , which picks action 1 near 51 percent.
Timed attacks perturb only decisive steps, so constant-noise scans miss them.
Poisoned rewards or doctored training states plant backdoors that wait for a trigger.
Clean scores overstate safety, so publish attacked scores at a stated epsilon beside them.
Masked gradients hide attacks without removing them, so test with black-box and transfer attackers.