From Cat Walks to Attention Heads: A Companion to MIT 6.S191's Sequence Modeling Lecture

Source: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention by Ava Amini · used as the primary text for this companion

Ava Amini's second lecture for MIT 6.S191 Introduction to Deep Learning (2026 edition) is a two-act play. Act one builds a memory: starting from the humble perceptron, it wires a neural network's output back into itself and arrives at the recurrent neural network. Act two dismantles that memory: it shows exactly why recurrence fails at scale, then replaces it with a mechanism that treats memory as search — attention — and lands on the transformer, the architecture inside GPT, BERT, and nearly every large model running today.

What this lecture is about

The lecture opens with a ball frozen in a 2D frame. Asked where it will go next, you can only guess. Shown the last few positions instead, the answer is obvious. That gap — between a snapshot and a trajectory — is the entire motivation for sequence modeling. Sequential data is everywhere: audio waveforms, text, medical signals, stock prices, biological sequences, video, weather. The task shapes vary too: a sequence can collapse to a single label, a single input can unfold into a sequence, or a sequence can map to another sequence.

Act one: neurons that remember

The build starts from lecture one's perceptron: inputs x₁ through x_m, each scaled by a weight, summed, and pushed through a nonlinearity to produce &yhat;. Stacked into a layer, perceptrons map multi-dimensional input to multi-dimensional output — but only for one moment in time. Applying that same feed-forward layer independently at each time step produces isolated predictions: x(0) knows nothing about x(1). Like the frozen ball, each frame is a guess.

The fix is a single extra wire. Give the layer an internal state, h, that is maintained by the network itself and passed from one time step to the next. Now the prediction at time t is a function of both the current input x_t and the previous state h_t−1. One network, one set of weights, reused at every step — that reuse is called weight sharing, and the loop is the recurrence relation that gives RNNs their name.

Unrolled over time, the RNN's math is two equations. The hidden state update mixes the previous state and the current input through two learned weight matrices and a tanh squashing function; a third learned matrix turns the state into a prediction:

Because the network emits a prediction at every step, you can score it at every step: the training loss is the sum of per-timestep losses across the whole sequence. In code, the loop is almost disappointingly simple — initialize h to zeros, feed words one at a time, update state, emit a prediction. The lecture shows a from-scratch TensorFlow cell before noting that in practice nobody writes this loop by hand: TensorFlow and PyTorch both ship built-in RNN layers, which the students use in the first software lab.

Four design criteria for sequence models

To make the requirements concrete, the lecture fixes on one task — next-word prediction, "the basis for basically all language models" — with the example sentence this morning I took my cat for a walk. Any architecture that wants this task has to satisfy four criteria.

Training across time — and how it breaks

RNNs are trained with ordinary backpropagation, extended over the unrolled computation graph: backpropagation through time (BPTT). Error flows from the final loss back through every time step to the beginning of the sequence. Every step of that backward trip multiplies by the same recurrent weight matrix W_hh — and that repeated multiplication is where things go wrong.

If the values of W_hh are large, gradients grow exponentially: exploding gradients, mitigated in practice by gradient clipping (capping gradient magnitude at a maximum). If the values are small, gradients shrink to nothing: the vanishing gradient problem, which is worse because it has no cheap fix. With error decaying at every step into the past, the network is biased toward short-term patterns and cannot learn dependencies that span many time steps — precisely criterion three.

The pioneering architectural fix, introduced by a group led by Jürgen Schmidhuber, was long short-term memory (LSTM): a more complex cell that controls the flow of information into and out of the state with learned gates, giving the gradient a highway that survives many steps. (The canonical citation is Hochreiter & Schmidhuber, Neural Computation 1997 — see the caveats.) A student question in the room gets a precise answer worth remembering: h₀, the initial state, is neither a learned parameter nor an input — it is an internal variable that depends on the weights and the data.

The application interlude is music. The students' first software lab is a many-output music-generation task, and the lecture's showcase example is Schubert's Unfinished Symphony: a recurrent model trained on Schubert's music was used to generate the missing third and fourth movements. The real 2019 Huawei project ran the model on a Mate 20 Pro phone NPU, with composer Lucas Cantor orchestrating the output for a Cadogan Hall performance — see the caveats for details.

Act two: three bottlenecks, one escape

For all its elegance, recurrence imposes three structural limits. First, the encoding bottleneck: everything the network knows about the past must be compressed into the single state vector h_t. Second, step-by-step processing is inherently sequential — not parallelizable — so long inputs are slow to handle. Third, those two together mean memory degrades fast as sequences grow.

So the lecture runs the thought experiment: eliminate recurrence entirely. The naive version — concatenate all time steps into one giant input vector and feed it to a dense network — fails twice over: it does not scale (the dense layer grows with sequence length), and it destroys order information, because nothing in a flat concatenation says which element came first. What is missing is a way to pick out which parts of the input matter. That is the hole attention fills.

Attention is a search problem

The intuition lands through YouTube. YouTube is a giant database; you type a query ("deep learning"). Each video has a title — a key — and the video content itself is the value. Search computes how well each key matches the query (a beautiful sea turtle: no; a six-year-old intro to deep learning: yes; Kobe Bryant's fadeaway: important but irrelevant), then returns the matching value — which, the lecturer notes, is of course their course. Self-attention is this same lookup, learned, and turned inward: within one sequence, which parts are most important to representing it?

Mechanically, one attention head does four things. The input first gets a positional encoding so that order survives the death of recurrence. Three separate learned layers transform that encoding into the query Q, key K, and value V matrices — same input, three different views. Similarity is computed as a scaled dot product between Q and K (the lecture notes this as a cosine-similarity-style measure; see caveats for the fine print). A softmax squashes the scores into attention weights between 0 and 1, and multiplying those weights by V extracts the features that mattered.

For the example he tossed the tennis ball to serve, the attention matrix lights up off the diagonal: "ball" and "tossed" score high because they are semantically related, not just positionally adjacent. And because every position is compared to every position at once, the whole operation parallelizes — the exact property recurrence lacked. Stack several heads in a multi-head attention layer and different heads can learn different patterns of importance simultaneously — the lecture's image is one head spotting Iron Man while another tracks the building he is about to hit and a third watches the alien ship behind him.

The payoff is the transformer: an architecture built from stacked attention heads, introduced by Vaswani et al. in the 2017 paper Attention Is All You Need. The T in GPT and BERT is literally this word. The closing minutes gesture at how far the idea has traveled — transformer LLMs in natural language, transformer-based models over protein and DNA sequences in biology, and vision transformers becoming a dominant architecture for image data (the subject of lecture three).

Why this lecture matters here

This lecture is unusually close to home. The Cat Chat / Cat-LLM stack documented across this site is built on GRUs — gated recurrent units, the 2014 sibling of the LSTM introduced by Cho et al. — trained at character level with exactly the BPTT machinery this lecture derives. The exploding-gradient clipping, the state-passing, the per-timestep loss: that is our training loop, running on Vulkan compute shaders instead of TensorFlow. Watch this lecture and you are watching the theory half of our stack; read the dev logs and you are watching its engineering half.

Chapter map

The video is chaptered; the eleven segments line up roughly with the two acts above. Timestamps in mm:ss.

Companion field guide

Quick reference for the terms that carry the lecture.

1. Sequence modeling — The paradigm of learning from time-resolved data, where each sample is an ordered series and predictions may depend on the whole history.
2. Recurrence relation — The rule h_t = f_W(h_t−1, x_t): the next state is a learned function of the previous state and the current input.
3. Hidden state (h_t) — The vector an RNN carries from step to step; its compressed memory of everything seen so far, and the source of the encoding bottleneck.
4. Weight sharing — Reusing the same W_hh, W_xh, W_hy matrices at every time step, which is what makes one network work over any sequence length.
5. Embedding — A learned numerical representation of tokens that places semantically related items near each other; the upgrade over sparse one-hot vectors.
6. Backpropagation through time (BPTT) — Gradient descent applied to the unrolled RNN graph, propagating error backward across time steps; involves repeated multiplication by W_hh.
7. Exploding gradient — When repeated multiplication by a large-valued matrix makes gradients grow without bound; handled by clipping gradients to a maximum norm.
8. Vanishing gradient — When repeated multiplication by a small-valued matrix shrinks gradients toward zero, starving early time steps of learning signal and crippling long-range memory.
9. LSTM — Long short-term memory (Hochreiter & Schmidhuber, 1997): an RNN cell with learned gates that regulate what enters, leaves, and persists in the state, creating a stable path for gradients.
10. GRU — Gated recurrent unit (Cho et al., 2014): a simpler two-gate alternative to LSTM with comparable long-range memory; the cell inside the Cat Chat models.
11. Encoding bottleneck — The constraint that an RNN must compress all past context into one fixed-size state vector; one of the three limits that motivate attention.
12. Positional encoding — Information added to input embeddings that encodes each token's position, restoring the order information lost when recurrence is removed.
13. Self-attention — Attention computed within a single sequence: every position scores its relevance to every other position in the same input.
14. Query / key / value — The three learned projections of each input position. Queries ask "what am I looking for," keys advertise "what do I contain," values carry "what do I contribute" if selected.
15. Scaled dot-product attention — softmax(Q·K^T/√d_k)·V: compare queries to keys by dot product, scale, normalize to weights, and mix values accordingly.
16. Multi-head attention — Running several attention heads in parallel and combining their outputs, so different heads can track different relation types at once.
17. Transformer — The attention-only architecture of Vaswani et al. (2017); the T in GPT and BERT. Dispenses with recurrence in favor of stacked self-attention.

Six study questions

1. Why does applying a feed-forward network independently at each time step fail the "predict the ball's trajectory" test, and what single addition fixes it?
2. The loss of an RNN is a sum over per-timestep losses. What does that assume about the task, and when might you only score the final step?
3. Explain why BPTT involves repeated multiplication by W_hh, and how that single fact produces both gradient pathologies.
4. An audience member asks whether h₀ is a parameter. What is the correct answer, and why does the distinction matter for training?
5. Why does concatenating all time steps into one input vector fail — which two of the four design criteria does it violate?
6. Trace "he tossed the tennis ball to serve" through one attention head: which matrix learns to make "ball" attend to "tossed," and where does softmax act?

Caveats and limitations

This companion is built from the lecture's caption track and external verification of the named results. Claims below are split into those verified against primary sources and those that are the lecturer's framing or need correction.

Related transmissions

The first half of this lecture is quite literally the theory under our GRU models, and the second half is the architecture they live alongside. Both show up as engineering all over the Frontier archive: char-level GRU pretraining, dual-stream GPU training, the Vulkan compute path those models run on, and a companion on the decoding side of LLM inference.