Core ML

Bidirectional RNNs

Run one recurrent chain forward and a second one backward over the same sequence, then combine them, so every position sees context from both directions at once.

A forward chain reads left to right and a backward chain reads right to left over the same three tokens, and their hidden states are concatenated at each position before anything downstream sees them

Why Does This Exist?

Take the sentence "The bank raised its rates" and ask a plain RNN what "bank" means by the time it processes that word. At that point in the sequence, the network has seen only "The" — nothing yet distinguishes a riverbank from a financial institution. The disambiguating word, "rates," doesn't arrive until three tokens later.

That's not a training problem. It's structural: a standard RNN's hidden state at position $t$ is a function of $x_1, \ldots, x_t$ only, by construction. However well it's trained, it cannot use information from $x_{t+1}$ onward, because that information hasn't been fed in yet when $h_t$ is computed. For generating text one token at a time, that's correct and unavoidable — the future genuinely doesn't exist yet. But for a huge class of tasks, the entire sequence is already sitting in memory before you start: tagging each word's part of speech, labeling named entities, scoring how well a candidate translation matches a source sentence. In all of those, refusing to look ahead is throwing away information that's right there.

Think of It Like This

Solving a crossword clue with both intersecting words visible

A crossword clue often can't be resolved from the letters coming before the blank alone — you need the crossing word too, which might run in from the other direction entirely. Someone who reads a clue left to right and refuses to look at any crossing entry is solving with one hand tied.

A bidirectional RNN is the two-handed version: one reader works through the sequence left to right, building up a "what came before" summary at every position; a second, completely independent reader works through the same sequence right to left, building a "what comes after" summary. Neither reader knows about the other while working — they run in parallel, on the same fixed input — and only afterward do you sit their two summaries for each position side by side.

How It Actually Works

Two independent chains

The diagram above shows the shape exactly. A forward chain computes $\overrightarrow{h}_1, \overrightarrow{h}_2, \ldots, \overrightarrow{h}_T$ the usual way, left to right, using any recurrent cell — vanilla, LSTM, or GRU. A backward chain, with its own separate set of weights, computes $\overleftarrow{h}_T, \overleftarrow{h}_{T-1}, \ldots, \overleftarrow{h}_1$ over the exact same input sequence, just processed in reverse order.

Crucially, these are two separate cells with two separate weight matrices — not one cell run twice. The forward cell never sees the backward cell's weights or outputs during its own pass, and vice versa. They can be computed in either order, or in parallel, because neither depends on the other's intermediate results.

Combining the two views

At every position $t$ , the two hidden states are concatenated into one vector:

h_t = [\overrightarrow{h}_t \, ; \, \overleftarrow{h}_t]

That combined $h_t$ is what any downstream layer — a classifier predicting a part-of-speech tag, say — actually reads. It carries a genuine summary of everything before position $t$ and everything after it, which is exactly the "bank … rates" disambiguation a unidirectional model can't perform at the word "bank" itself.

The tradeoff that comes with it

This context comes at a specific, unavoidable cost: the entire input sequence must be available before you can compute anything. The backward chain's very first computation, $\overleftarrow{h}_T$ , needs the last token in the sequence. That rules bidirectional RNNs out for any task where you're generating output one step at a time and the future genuinely doesn't exist yet — language generation, real-time transcription, anything autoregressive. It's the right tool exactly when you called out in the opening: the whole sequence is already sitting in memory, and you're labeling or scoring it, not producing it token by token.

Show Me the Code

Two independent RNN cells over the same three-token input, concatenated position by position — the entire mechanism in one loop.

1import numpy as np2
3
4def rnn_step(x_t: np.ndarray, h_prev: np.ndarray, w: np.ndarray) -> np.ndarray:5    z = np.concatenate([x_t, h_prev])6    return np.tanh(w @ z)7
8
9def bidirectional_pass(10    sequence: list[np.ndarray], w_fwd: np.ndarray, w_bwd: np.ndarray, hidden_size: int11) -> list[np.ndarray]:12    fwd_states, h = [], np.zeros(hidden_size)13    for x_t in sequence:                                  # forward: left to right14        h = rnn_step(x_t, h, w_fwd)15        fwd_states.append(h)16
17    bwd_states, h = [], np.zeros(hidden_size)18    for x_t in reversed(sequence):                        # backward: right to left19        h = rnn_step(x_t, h, w_bwd)20        bwd_states.append(h)21    bwd_states.reverse()                                   # realign to left-to-right order22
23    return [np.concatenate([f, b]) for f, b in zip(fwd_states, bwd_states)]24
25
26rng = np.random.default_rng(0)27tokens = [np.array([1.0, 0.0]), np.array([0.0, 1.0]), np.array([1.0, 1.0])]28w_f, w_b = rng.normal(scale=0.4, size=(3, 5)), rng.normal(scale=0.4, size=(3, 5))29outputs = bidirectional_pass(tokens, w_f, w_b, hidden_size=3)30print(outputs[1].shape)  # -> (6,) — forward h_2 concatenated with backward h_2

outputs[1] — the representation for the middle token — genuinely depends on all three tokens: the forward half saw tokens 1 and 2, the backward half saw tokens 2 and 3.

Watch Out For

Using a bidirectional layer where generation is the task

A bidirectional layer needs the whole sequence up front, and it's easy to reach for one out of habit inside an autoregressive decoder — the part of a model generating text token by token. That's a structural contradiction: at generation time, "future" tokens don't exist yet, so a backward pass has nothing valid to read. This isn't a subtle bug that trains poorly; frameworks generally won't let you run a backward RNN pass over a sequence you haven't finished producing. Reserve bidirectional layers for encoders and classifiers over already-complete input, and keep decoders unidirectional.

Assuming double the hidden size means double the parameters, only

A bidirectional layer roughly doubles both the parameter count (two full sets of recurrent weights) and the compute per step (two passes instead of one), and it also doubles the output dimensionality that every downstream layer has to accept. Forgetting that last part — sizing a classifier's input for a single-direction hidden size when the actual input is the concatenated pair — is a shape mismatch that surfaces immediately, but it's worth planning for rather than discovering.

The Quick Version

A standard RNN's hidden state at position $t$ can only depend on tokens up to $t$ — the future literally hasn't been fed in yet.
A bidirectional RNN runs two independent chains, one forward and one backward, each with its own weights.
The two hidden states at each position are concatenated, giving every position a summary of both what came before and what comes after.
The requirement is the whole sequence available up front — which rules bidirectional layers out for autoregressive generation.
Parameter count and per-step compute both roughly double compared to a single-direction chain.