Letters as points

In the last chapter a network looked at three letters and guessed the fourth. But before doing any sums it did something odd: it swapped every letter for a list of sixteen numbers. Nobody chose those numbers. The network picked them itself, along with all its other weights. So what do they say?

Sixteen numbers are impossible to draw, so we trained the same kind of network again and allowed it only two numbers per letter. Two numbers are the coordinates of a point on a plane, which means we can put every letter on a map and see where the network placed it.

The map the network drew

Below are all 97 symbols our letter models know: lower- and upper-case letters, digits, punctuation, the space, the newline and the start-of-text mark. Each one sits at the point given by its two numbers after training. The colours by group are ours; the network knew nothing about groups.

Look closely and order jumps out at you. The vowels a, e, i, o, u have gathered in one corner, with their capitals A, I, U, O, E beside them. The full stop, the exclamation and question marks, the colon and the semicolon keep together on the opposite side. Space, newline and the start mark huddle in a little group: to the network they are one and the same event, "a word ended here". Many capitals stand next to their lower-case twins: t and T, h and H, n and N.

The rare symbols, like ^, {, ~ and |, crowd the middle in no particular order. We checked: they have barely moved from the random spots where they started before training. They almost never appear in the text, so the network hardly ever touched them.

Why did the vowels end up together? Nobody told the network they were vowels. But they lead similar lives: they are usually preceded and followed by consonants. It pays the network to make letters that behave alike look alike, because then whatever it learns about a automatically works for o. Training kept nudging such letters towards each other until they became neighbours.

Now check that the network really sees nothing but coordinates. The prediction panel holds the context "␣th": a space, t, h. After it the network expects e with about 76% probability, since the is the likely word. Now drag the h over to the vowels, close to a. The prediction flips: the network starts expecting the consonants m and l, as it would after "␣ta". Nothing inside the network changed; you moved two numbers. Drag the h to the full stop instead, and the network decides the sentence is over and expects a newline.

A number, a row of switches, a point

Why did we need coordinates at all? In chapter 1 every symbol got a number: space is 2, a is 67, b is 68, z is 92. Why not feed that number straight into the network?

Because the network multiplies its inputs by weights and adds them up: it treats them as quantities. It would think b is "a bit more" than a and that the tilde, number 96, is a "huge" letter. We handed out the numbers in ASCII order, and their size means nothing.

The next idea is one-hot encoding. Picture a row of 97 switches, one per symbol; for a, flip on switch 67 and nothing else. That gives a vector of 97 numbers: a single one and zeros everywhere else. No false "bigger" or "smaller"; every symbol is treated the same.

Too much so, in fact. Any two one-hot vectors are exactly as far apart as any other two: a is as unlike A as it is unlike a comma. Nothing the network learns about a tells it anything about A, so it has to learn every letter from scratch. And the vector is long and empty: 96 zeros to carry one 1.

An embedding is the third way. Give each symbol a short vector of $d$ ordinary numbers (our map has $d = 2$, last chapter's network 16, Sprout 384) and let training move those numbers around. All the vectors live in one table $E$ of size $97 \times d$: row 67 is the vector for a. To turn a letter into a vector, you just fetch its row.

How alike are two vectors?

We keep saying "similar letters ended up close". To work with that properly we need a measure of similarity: one number that says how much two vectors point the same way. There is one, and it is the most important operation in this whole course: the dot product. Multiply the coordinates pair by pair and add:

$$a \cdot b = a_1 b_1 + a_2 b_2 + \dots + a_d b_d.$$

In two dimensions that's just two products added together. Play with the arrows and watch when the number is large, when it is zero and when it goes negative.

The pattern: arrows pointing the same way give a large positive number; at right angles, zero; pointing apart, a negative one. But try "b twice as long": same direction, double the number ("b half as long" halves it). The dot product feels both direction and length.

If only direction matters, divide by the lengths. That gives cosine similarity, a number between −1 and 1:

$$\cos\theta = \frac{a \cdot b}{|a|\,|b|}, \qquad |a| = \sqrt{a \cdot a} = \sqrt{a_1^2 + a_2^2 + \dots}$$

You will meet this move everywhere. The neuron from the last chapter is a dot product of its input with its weights, plus a bias. Attention in chapter 7 uses dot products to decide which words matter. Even Sprout's very last step, a probability for each of 8,192 tokens, starts as 8,192 dot products.

Matrix multiplication is many dot products

A neuron takes an input and computes one dot product. A layer of 256 neurons computes 256 dot products with the same input. And during training the network handles not one three-letter window but hundreds at once. Hundreds of inputs times hundreds of neurons is tens of thousands of dot products, far too many to write one by one. There is a single operation for the job: matrix multiplication.

A matrix is a rectangular table of numbers. In the product $C = A B$, cell $c_{ij}$ is the dot product of row $i$ of $A$ with column $j$ of $B$:

$$c_{ij} = \sum_{k} a_{ik}\, b_{kj}.$$

Hence the rule for shapes: the rows of $A$ must be as long as the columns of $B$ are tall. An $n \times k$ matrix times a $k \times m$ one gives $n \times m$. Here is what that looks like:

Now switch the widget to "One-hot × table". On the left is a three-letter word written as one-hot rows; on top is a tiny embedding table (real coordinates from the map). Compute the cells: in each dot product only one term survives, the one under the 1, and everything else is multiplied by zero. What comes out is simply the rows of the table.

So "fetch a row of the table" and "multiply a one-hot vector by the table" are the same thing. That's why real code never multiplies by vectors full of zeros; it fetches the row by number, E[id]. Thousands of times faster, and mathematically identical. Here is the same idea in numpy:

import numpy as np # four letters and their coordinates from the map above letters = ['a', 'e', 't', 'h'] E = np.array([[ 3.70, -2.71], [ 0.78, -1.81], [-1.79, -2.05], [-0.73, 1.24]]) ids = [2, 3, 1] # "the" one_hot = np.eye(4)[ids] # three rows with a single 1 each print(one_hot) print(one_hot @ E) # multiply by the table... print(E[ids]) # ...or just take the rows: the same numbers def cosine(u, v): return u @ v / (np.linalg.norm(u) * np.linalg.norm(v)) print(' ' + ' '.join(letters)) for i, x in enumerate(letters): print(x, ' '.join(f'{cosine(E[i], E[j]):+.2f}' for j in range(4)))

In numpy and PyTorch the @ sign means matrix multiplication. The cosine table says what the map showed: a and e have a similarity of +0.86, while h points almost opposite to both vowels.

Tensors and batches

One more word to know: tensor. It is just a table of numbers with any number of dimensions: a single number has none, a vector one, a matrix two. A batch of $B$ windows of 3 letters is a tensor of ids with shape $(B, 3)$. Fetching embeddings gives every letter 16 numbers, and the shape becomes $(B, 3, 16)$. We glue each window's three vectors into one long one, $(B, 48)$, and multiply by the hidden layer's weight matrix. Let's follow the shapes all the way from ids to logits:

import numpy as np rng = np.random.default_rng(0) B, T, d, H, V = 4, 3, 16, 256, 97 # batch, window, embedding, hidden, vocabulary E = rng.normal(size=(V, d)) # the embedding table: a row per symbol W1 = rng.normal(size=(H, T * d)) * 0.1 # nn.Linear stores weights as (out, in) W2 = rng.normal(size=(V, H)) * 0.1 x = rng.integers(0, V, size=(B, T)) # 4 windows of 3 letter ids e = E[x] # look-up: (4, 3, 16) flat = e.reshape(B, T * d) # glue each window's vectors: (4, 48) hid = np.tanh(flat @ W1.T) # 4 × 256 dot products in one go logits = hid @ W2.T # (4, 97): a score for every next letter for name, a in [('x', x), ('e', e), ('flat', flat), ('hid', hid), ('logits', logits)]: print(f'{name:7s}', a.shape)

The single line flat @ W1.T is $4 \times 256 = 1{,}024$ dot products of 48 terms each. GPUs and CPUs do such multiplications astonishingly fast, and all of modern deep learning rests on being able to write almost any job as a matrix multiplication.

What do you get when you multiply the one-hot row for "t" by the embedding table?

The sum of all the table's rows The table row that belongs to "t" The dot product of "t" with every other letter

A one-hot row has a single 1 and zeros elsewhere. In every dot product only one term survives, so the result is exactly the row for "t". That's why the multiplication is replaced by a plain E[id].

The same network in PyTorch

So far we have written everything by hand. The course's real models are written in PyTorch, a library that does three things: it stores tensors, multiplies them fast (on a GPU too), and computes gradients by itself, like our Value from the last chapter, but for tensors of any size. Here is the class behind every letter network in the course, from the map at the top of this chapter to the model at the end. It comes from snapshots.py unchanged:

class MLP(nn.Module): def __init__(self, vocab, context, d_embed, hidden): super().__init__() self.context = context self.embed = nn.Embedding(vocab, d_embed) dims = [context * d_embed] + hidden self.hidden = nn.ModuleList(nn.Linear(a, b) for a, b in zip(dims, dims[1:])) self.out = nn.Linear(dims[-1], vocab) def forward(self, x): # x: (batch, context) h = self.embed(x).flatten(1) for layer in self.hidden: h = torch.tanh(layer(h)) return self.out(h)

Three details, all familiar by now:

• nn.Embedding(vocab, d_embed) is the table $E$, 97 × d_embed. Calling self.embed(x) fetches rows by number, our E[x]. During training the gradient only reaches the rows of letters that were in the batch.
• nn.Linear(a, b) is a layer of b neurons with a inputs each: a $b \times a$ weight matrix and a bias vector. Calling it computes x @ W.T + bias, a batch of dot products.
• flatten(1) glues a window's letter vectors into one: shape $(B, 3, 16)$ becomes $(B, 48)$.

The map at the top of the chapter is MLP(97, 3, 2, [128]): three letters of two numbers each and a hidden layer of 128 neurons. The last chapter's network is MLP(97, 3, 16, [256]). The model you'll meet at the end is MLP(97, 8, 24, [512, 512]): eight letters of 24 numbers and two hidden layers.

Learning in batches

How do we train such a network? Same idea as chapter 3: compute the loss, find the gradient, take a step downhill. The question is how many examples to average the loss over before each step.

We could use all of them. But our letter corpus holds tens of millions of windows, and one small step would mean running through every one. We could use a single example: steps would be quick, but each would lurch its own way, because one window says nothing about the language as a whole. The happy medium is the minibatch: take a few hundred random windows, average the loss over them and take a step. The direction isn't perfect, but it is right on average, and a batch takes a fraction of a second, thanks to matrix multiplication.

Here is the real training loop of every letter model in the course, also from snapshots.py:

def train_mlp(name, ids, vocab, context, d_embed, hidden, args, lr=3e-3, steps=None, batch=512): torch.manual_seed(0) data = windows(ids, context) n_val = len(data) // 50 val, tr = data[:n_val], data[n_val:] model = MLP(vocab, context, d_embed, hidden) opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.01) steps = steps or args.steps t0 = time.time() log = [] for step in range(steps + 1): if step % 500 == 0 or step == steps: with torch.no_grad(): idx = torch.randint(0, len(val), (8192,), generator=torch.Generator().manual_seed(0)) vb = val[idx] vl = F.cross_entropy(model(vb[:, :-1]), vb[:, -1]).item() log.append({'step': step, 'val': round(vl, 4)}) print(f'{name} step {step} val {vl:.4f} ({time.time() - t0:.0f}s)', flush=True) if step == steps: break b = tr[torch.randint(0, len(tr), (batch,))] loss = F.cross_entropy(model(b[:, :-1]), b[:, -1]) opt.zero_grad(set_to_none=True) loss.backward() opt.step() for g in opt.param_groups: g['lr'] = lr * (0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * step / steps))) return model, log

Let's read it from the top.

1. windows cuts the text into windows: a fixed number of letters of context plus the next letter, which is the right answer. The first 2% of the windows (len(data) // 50) are put aside; more on that below.
2. Each step takes 512 random windows: b[:, :-1] is the context, b[:, -1] the answers.
3. F.cross_entropy is our old friend from chapter 2: a softmax over the logits, minus the log of the right answer's probability, averaged over the batch. One function instead of three lines.
4. loss.backward() is chapter 4's backward pass: PyTorch finds the gradient for each of hundreds of thousands of parameters, rows of the embedding table included.
5. opt.step() takes the step. Instead of plain gradient descent this uses AdamW, a smarter way of choosing each parameter's step; chapter 10 takes it apart. The last two lines gently lower the learning rate towards the end.

The chapter's final model took 20,000 such steps, so it saw about 10 million windows. On a laptop with an M4 Pro that took about three minutes.

An exam on unseen text

Why put 2% of the windows aside? Because the loss on the training text can lie. A network can get very good at predicting exactly the windows it has seen, by memorising them, while understanding nothing about the language. There is only one honest test: give it text it has never seen and measure the loss there. The part set aside is the validation set; the rest is the training set.

It's easiest to see for yourself. Below is a tiny network with two numbers per letter, learning right in your browser on a slice of real stories from our corpus. On the left is its letter map; on the right, two curves: the loss on the training text and on the validation text. Start with 3,000 letters, then try 400 and the whole text.

On the whole text both curves go down together, and one by one the vowels drift away from the huddle of consonants: the map assembles itself as you watch. With 400 letters something else happens. The training loss drops far lower than on the big text, while the validation loss dips a little, then turns and climbs, until it is worse than blind guessing. The network has 1,764 parameters and fewer than 400 examples: memorising every window is easier than finding rules. This is overfitting. With 3,000 letters you can watch it set in: the validation loss falls for a while, bottoms out and slowly creeps up again.

Two practical rules follow. One: judge a model only by its validation loss. Two: the best cure for overfitting is more data. Sprout, with 17 million parameters, read 330 million tokens during training, less than a quarter of the corpus, so almost everything it read was new to it, and there was nothing to cram.

The training loss keeps falling while the validation loss rises. What's going on?

The learning rate is too small The model is memorising the training examples instead of general patterns The validation text is broken

That's overfitting. The model remembers what it has seen better and better and handles new text worse and worse. More data helps, as does a smaller model, or stopping early where the validation loss was lowest.

A wider window, a smarter model

Now we have everything we need to make the model stronger. The obvious move is to let it see more. The bigram of chapter 1 looked at one letter; last chapter's network at three. Let's take eight, give each letter 24 numbers and add a second hidden layer. We measured all these models on the same text, which none of them saw in training: 4.9 million letters.

Model	Letters seen	Parameters	Nats per letter
blind guess over 97 symbols	0	0	4.575
bigram, chapter 1	1	9,409	2.364
the map, 2 numbers a letter	3	13,603	1.874
network, chapter 4	3	39,025	1.479
network, this chapter	8	413,561	1.103

Every row is a real step forward. Even the tiny map with two numbers per letter beats the bigram by a mile: three letters of context help, and so does letting similar letters share what they learn. More numbers per letter and more neurons bring 1.479. Eight letters bring 1.103. In terms of perplexity (chapter 2), that is a fall from $e^{2.364} \approx 10.6$ "equally likely options" per letter for the bigram to $e^{1.103} \approx 3.0$.

See the difference for yourself: the three models write at the same time, from the same start, at the same temperature.

The bigram glues together word-shaped scraps. The three-letter network already writes lots of real short words. The eight-letter one strings together whole phrases from stories, like "there was a little", "One day," or "felt a", but the meaning still snaps every few words. No wonder: eight letters are a word and a half. Everything before them is invisible to the model.

Why not a window of 500 letters?

The first layer of such a network takes all the window's vectors laid end to end. For our model that is 8 × 24 = 192 numbers, and the first layer holds 192 × 512 = 98,304 weights. A 500-letter window would need 62 times as many weights in the first layer alone, most of them learning from rare cases.

Worse, every position in the window has weights of its own: a "t" in position one and the same "t" in position five go through different numbers. Whatever the network learned about "the" at the start of the window has to be learned again for every shift. That is wasteful, and there are two ways out. One is to make the pieces bigger so more text fits in the same window: that's tokens, the next chapter. The other is a mechanism that treats every position the same way and decides for itself where to look: attention, chapter 7.

Sprout right now

Sprout still reads letter by letter, but now it sees eight at once and knows which letters are alike. It writes almost real English phrases, although after a couple of words it forgets how it started. Its surprise has dropped from the bigram's 2.364 nats per letter to 1.103. In the next chapter we stop feeding it letters: the window keeps its eight slots, but each slot will hold a whole piece of a word.