RNNs & Sequence Models

Start here — why order changes everything

Everything you've built so far took in a fixed lump of data — one image, one row of a spreadsheet — and produced one answer. But an enormous amount of the world doesn't arrive as a lump. It arrives one piece at a time, in an order that matters:

• Words in a sentence: "dog bites man" and "man bites dog" use identical words and mean opposite things.
• A heartbeat trace, an audio waveform, a stock price, a sensor reading — a number per moment.
• Frames of a video: the same frames shuffled are a different event.

A sequence is any data like this. Each position in it is a timestep, usually written with the letter $t$ (for time).

Why not just use a normal network?

The obvious hack is: take the last 5 words, glue them into one long input vector, feed that to an ordinary network. This is called an $n$-gram or fixed-window model, and it genuinely works — a bit. But it breaks in three ways:

1. The window is a hard wall. Anything older than 5 words is invisible. "The keys I left on the kitchen counter this morning after the argument about the dog are ___" — the word you need ("keys") is far outside any reasonable window.
2. Length is fixed. You have to pick the window size up front, and pad or truncate everything to fit.
3. Nothing is shared across positions. The network learns "what does a verb in slot 3 mean" separately from "what does a verb in slot 4 mean." It has to relearn the same grammar once per position, wasting both parameters and data.

The core idea — a loop with a memory

Here is the whole invention, in one sentence: use one small network, feed it the sequence one item at a time, and let it pass a note to itself.

The note is called the hidden state, written $h_t$, and the loop itself — network feeding its own output back into its own input — is the recurrence. A network built this way is a RNN.

Do one by hand

Let's make everything as small as it can possibly be: the hidden state is a single number, and so is each input. The network has two weights — one for "how much of my old note do I keep" ($w_h$) and one for "how much does the new input matter" ($w_x$) — and it squashes the result with $\tanh$ so the note can never blow up past $\pm 1$.

Here is that fade, plotted, next to the same network with a stickier memory ($w_h = 0.95$ — keep 95% each step). Same single ping at step 1, same silence afterwards:

Run it yourself — the code reproduces the hand calculation exactly, and you can change the weights:

Backpropagation through time — how a loop learns

You now have a network with a loop. How do you take a gradient through a loop?

This is called BPTT, and the drawing move is called unrolling.Deep Learning Book

Why plain RNNs forget — vanishing and exploding gradients

That knife-edge is the vanishing gradient problem and its twin, the exploding gradient problem — the reason a plain RNN, in practice, has a memory of maybe ten or twenty steps.Pascanu 2013

See the two failure modes on one chart. The vertical axis is logarithmic, so a straight line is exponential growth or decay:

LSTM — build a memory you write to on purpose

The plain RNN's flaw is that it overwrites its entire note at every single step. Every memory must survive by being re-copied through a matrix multiply and a squash, thousands of times. Nothing survives that.

The fix, from 1997, is startlingly direct: give the network a separate memory that is left alone by default, and let it learn small, explicit decisions about what to erase, what to write, and what to reveal.Hochreiter 1997

The whiteboard is the cell state $c_t$; each tap is a gate. A network built this way is an LSTM.

Why gates are made of sigmoids

A gate has to output "how much gets through", so it needs to live strictly between 0 and 1. That is precisely what the sigmoid function does — and it's why every gate in every gated architecture is a sigmoid, while the content being written is a $\tanh$ (which spans $-1$ to $+1$, so a write can push a value up or down).

Play with the two curves — set the input low, high, and near zero, and watch the sigmoid act as a soft switch while $\tanh$ acts as a signed, squashed value:

Watch the difference over 100 steps. Both start holding the value 1.0 and receive no new input; one is a plain RNN hidden state, the other an LSTM cell state with its forget gate open:

GRU — the same trick with fewer moving parts

The LSTM works, but it carries two states ($\mathbf{h}$ and $\mathbf{c}$) and four sets of weights. In 2014 a slimmer variant appeared alongside the encoder–decoder architecture: the GRU.Cho 2014

The shapes sequence problems come in

Once you have a recurrent layer, you can wire it up in several ways depending on what goes in and what comes out:

Two more wiring tricks are worth knowing:

• Stacking. Feed one recurrent layer's hidden states into another as inputs. Deeper stacks build more abstract features over time, exactly like stacking convolutions in a CNN. Two to four layers is typical.
• Bidirectional. Run one RNN left-to-right and another right-to-left, then concatenate their states at each position. Now every position sees the whole sequence, past and future. This is superb for understanding tasks (tagging, classification) and impossible for generation — you can't peek at future words you haven't written yet.

Seq2seq, the bottleneck, and the road to attention

The great application was translation. Sutskever, Vinyals and Le showed you could stack two LSTMs: an encoder reads the source sentence and compresses it into a single fixed-length vector, and a decoder reads that vector and generates the target sentence one word at a time.Sutskever 2014

The fix, in 2015, was to stop forcing everything through one vector. Keep all the encoder's hidden states, and let the decoder, at each output word, compute a set of weights over them and take a weighted average — looking hardest at the source words that matter for the word it's about to produce.Bahdanau 2015 Those weights are an alignment, and this mechanism is attention.

Here is what such an alignment looks like. Each row is one output word choosing where to look; brighter means "I am paying attention here". English and Spanish put adjectives on opposite sides of the noun, and the model has to cross over:

Why Transformers replaced RNNs

Attention was invented as a patch for the seq2seq bottleneck. Then came the observation that changed everything: if attention lets any output position look directly at any input position, why keep the recurrence at all? Drop it, and you get the Transformer.

The decisive advantage is not accuracy — it's parallelism. An RNN must compute $\mathbf{h}_1$ before $\mathbf{h}_2$ before $\mathbf{h}_3$: a sequence of length 1000 is 1000 unavoidably sequential operations, which is a terrible fit for a GPU that wants to do thousands of things at once. A Transformer computes all positions simultaneously. That single property is what made training on trillions of tokens practical, and it is the reason essentially every large language model today is a Transformer rather than an LSTM. Continue that story in Attention & Transformers.

In real code

In practice you never hand-roll the loop. Here's the PyTorch version — the same architecture, three lines of definition:

import torch
import torch.nn as nn

class SequenceClassifier(nn.Module):
    def __init__(self, vocab_size, emb=128, hidden=256, n_classes=2):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, emb)
        # batch_first=True -> tensors are (batch, time, features)
        # bidirectional    -> reads left-to-right AND right-to-left (fine: we classify, not generate)
        self.lstm = nn.LSTM(emb, hidden, num_layers=2, batch_first=True,
                            bidirectional=True, dropout=0.2)
        self.head = nn.Linear(hidden * 2, n_classes)   # *2 for the two directions

    def forward(self, tokens):
        x = self.embed(tokens)                 # (B, T, emb)
        out, (h_n, c_n) = self.lstm(x)         # out: every timestep's hidden state
        last = out[:, -1, :]                   # many-to-one: read the final step
        return self.head(last)

model = SequenceClassifier(vocab_size=20_000)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)

for tokens, labels in loader:
    opt.zero_grad()
    loss = nn.functional.cross_entropy(model(tokens), labels)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)   # never optional for RNNs
    opt.step()

Practice — and how to make it stick

1. By hand: carry out three LSTM steps with $f = [0.99, 0.1]$, $i = [0.01, 0.9]$, a candidate of $[0.5, -0.5]$, and $c_0 = [1, 1]$. Which of the two slots is behaving like long-term memory, and which like a scratchpad?
2. From scratch: implement a character-level RNN in NumPy and train it on a page of text, then sample from it. Karpathy's write-up is the canonical walkthrough of exactly this.Karpathy — RNNs
3. Measure the forgetting: build the copy task — show the model a random 8-token code, then $N$ steps of noise, then ask it to reproduce the code. Sweep $N$ and plot accuracy for a plain RNN vs an LSTM. You will see the plain RNN's cliff.
4. Break it on purpose: train an LSTM with gradient clipping turned off and a high learning rate. Watch the loss go NaN, then turn clipping on and watch it survive.

Start with the lab below — edit and run it right here, and if you get stuck or hit an error, ask Ada on the right: she can see your code and your terminal output.

Next, see what happened when the field dropped the recurrence entirely: Attention & Transformers.