Letters as points

Before predicting a fourth character, our network replaces each of the preceding three with a vector of sixteen numbers. Why those particular numbers? Their values were adjusted during training along with the other parameters. Let’s examine this representation and see what patterns in the text it reveals.

We cannot show a sixteen-dimensional vector directly as a point on a plane without losing information. For this experiment, we trained a separate network with narrower embeddings: just two numbers per character. We can use those numbers as coordinates. The map shows this network’s actual vectors, not a projection of the sixteen-dimensional vectors from the last chapter.

The map the network drew

Below are all 97 symbols our letter models know: lower- and upper-case letters, digits, punctuation, the space, the newline and the start-of-text mark. Each one sits at the point given by its two numbers after training. The colours by group are ours; the network knew nothing about groups.

Look closely and order jumps out at you. The vowels a, e, i, o, u have gathered in one corner, with their capitals A, I, U, O, E beside them. The full stop, the exclamation and question marks, the colon and the semicolon keep together on the opposite side. Space, newline and the start mark huddle in a little group: to the network they are one and the same event, "a word ended here". Many capitals stand next to their lower-case twins: t and T, h and H, n and N.

The rare symbols, like ^, {, ~ and |, crowd the middle in no particular order. We checked: they have barely moved from the random spots where they started before training. They almost never appear in the text, so the network hardly ever touched them.

We never told the network which letters were vowels. Groups emerge because characters occur in similar contexts and call for similar continuations. Nearby vectors let later layers handle those cases in similar ways. But the loss function contains no instruction to “move the vowels together”: the arrangement emerges while embeddings and other weights are trained jointly. Another run could rotate the map or arrange it differently.

Now check that the network really sees nothing but coordinates. The prediction panel holds the context "␣th": a space, t, h. After it the network expects e with about 76% probability, since the is the likely word. Now drag the h over to the vowels, close to a. The prediction flips: the network starts expecting the consonants m and l, as it would after "␣ta". You changed two learned embedding parameters; the other weights stayed the same. Drag the h to the full stop instead, and the network decides the sentence is over and expects a newline.

A number, a row of switches, a point

Why did we need coordinates at all? In chapter 1 every symbol got a number: space is 2, a is 67, b is 68, z is 92. Why not feed that number straight into the network?

Because the network multiplies its inputs by weights and adds them up: it treats them as quantities. It would think b is "a bit more" than a and that the tilde, number 96, is a "huge" letter. We handed out the numbers in ASCII order, and their size means nothing.

The next idea is one-hot encoding. Picture a row of 97 switches, one per symbol; for a, flip on switch 67 and nothing else. That gives a vector of 97 numbers: a single one and zeros everywhere else. No false "bigger" or "smaller"; every symbol is treated the same.

Different one-hot vectors are equally far apart, so the representation itself says nothing about the similarity of “a” and “A”. A network receiving one-hot vectors can still learn shared patterns, though. Multiplying those vectors by a trainable matrix gives us embeddings. A short, dense vector is simply more convenient to store and pass to later layers than 97 numbers containing just one nonzero entry.

An embedding is the third way. Give each symbol a short vector of $d$ ordinary numbers (our map has $d = 2$, last chapter's network 16, Sprout 384) and let training move those numbers around. All the vectors live in one table $E$ of size $97 \times d$: row 67 is the vector for a. To turn a letter into a vector, you just fetch its row.

How alike are two vectors?

On a map we can measure distances between points; with vectors, it is also useful to compare directions. Start with the dot product: multiply matching coordinates and add the results. It depends on both the angle between the vectors and their lengths, so it is not yet a measure of direction alone:

$$a \cdot b = a_1 b_1 + a_2 b_2 + \dots + a_d b_d.$$

In two dimensions that's just two products added together. Play with the arrows and watch when the number is large, when it is zero and when it goes negative.

The pattern: arrows pointing the same way give a large positive number; at right angles, zero; pointing apart, a negative one. But try "b twice as long": same direction, double the number ("b half as long" halves it). The dot product feels both direction and length.

To compare only the directions of two nonzero vectors, divide their dot product by the product of their lengths. This gives cosine similarity, between −1 and 1. A zero vector has no direction, so the formula is undefined for it:

$$\cos\theta = \frac{a \cdot b}{|a|\,|b|}, \qquad |a| = \sqrt{a \cdot a} = \sqrt{a_1^2 + a_2^2 + \dots}$$

You will meet this move everywhere. The neuron from the last chapter is a dot product of its input with its weights, plus a bias. Attention in chapter 7 uses dot products to decide which words matter. Even Sprout's very last step, a probability for each of 8,192 tokens, starts as 8,192 dot products.

Matrix multiplication is many dot products

A neuron takes an input and computes one dot product. A layer of 256 neurons computes 256 dot products with the same input. And during training the network handles not one three-letter window but hundreds at once. Hundreds of inputs times hundreds of neurons is tens of thousands of dot products, far too many to write one by one. There is a single operation for the job: matrix multiplication.

A matrix is a rectangular table of numbers. In the product $C = A B$, cell $c_{ij}$ is the dot product of row $i$ of $A$ with column $j$ of $B$:

$$c_{ij} = \sum_{k} a_{ik}\, b_{kj}.$$

Hence the rule for shapes: the rows of $A$ must be as long as the columns of $B$ are tall. An $n \times k$ matrix times a $k \times m$ one gives $n \times m$. Here is what that looks like:

Now switch the widget to "One-hot × table". On the left is a three-letter word written as one-hot rows; on top is a tiny embedding table (real coordinates from the map). Compute the cells: in each dot product only one term survives, the one under the 1, and everything else is multiplied by zero. What comes out is simply the rows of the table.

Multiplying a one-hot vector by a table selects one row. Code therefore uses an indexed lookup, E[id]. It avoids creating a large vector of zeros and doing unnecessary multiplications. The mathematical result is identical; the speed improvement depends on the dimensions and implementation. Let’s check the equivalence in NumPy:

import numpy as np # four letters and their coordinates from the map above letters = ['a', 'e', 't', 'h'] E = np.array([[ 3.70, -2.71], [ 0.78, -1.81], [-1.79, -2.05], [-0.73, 1.24]]) ids = [2, 3, 1] # "the" one_hot = np.eye(4)[ids] # three rows with a single 1 each print(one_hot) print(one_hot @ E) # multiply by the table... print(E[ids]) # ...or just take the rows: the same numbers def cosine(u, v): return u @ v / (np.linalg.norm(u) * np.linalg.norm(v)) print(' ' + ' '.join(letters)) for i, x in enumerate(letters): print(x, ' '.join(f'{cosine(E[i], E[j]):+.2f}' for j in range(4)))

In numpy and PyTorch the @ sign means matrix multiplication. The cosine table says what the map showed: a and e have a similarity of +0.86, while h points almost opposite to both vowels.

Tensors and batches

One more word to know: tensor. It is just a table of numbers with any number of dimensions: a single number has none, a vector one, a matrix two. A batch of $B$ windows of 3 letters is a tensor of ids with shape $(B, 3)$. Fetching embeddings gives every letter 16 numbers, and the shape becomes $(B, 3, 16)$. We glue each window's three vectors into one long one, $(B, 48)$, and multiply by the hidden layer's weight matrix. Let's follow the shapes all the way from ids to logits:

import numpy as np rng = np.random.default_rng(0) B, T, d, H, V = 4, 3, 16, 256, 97 # batch, window, embedding, hidden, vocabulary E = rng.normal(size=(V, d)) # the embedding table: a row per symbol W1 = rng.normal(size=(H, T * d)) * 0.1 # nn.Linear stores weights as (out, in) W2 = rng.normal(size=(V, H)) * 0.1 x = rng.integers(0, V, size=(B, T)) # 4 windows of 3 letter ids e = E[x] # look-up: (4, 3, 16) flat = e.reshape(B, T * d) # glue each window's vectors: (4, 48) hid = np.tanh(flat @ W1.T) # 4 × 256 dot products in one go logits = hid @ W2.T # (4, 97): a score for every next letter for name, a in [('x', x), ('e', e), ('flat', flat), ('hid', hid), ('logits', logits)]: print(f'{name:7s}', a.shape)

The single line flat @ W1.T is $4 \times 256 = 1{,}024$ dot products of 48 terms each. GPUs and CPUs do such multiplications astonishingly fast, and all of modern deep learning rests on being able to write almost any job as a matrix multiplication.

What do you get when you multiply the one-hot row for "t" by the embedding table?

The sum of all the table's rows The table row that belongs to "t" The dot product of "t" with every other letter

A one-hot row has a single 1 and zeros elsewhere. In every dot product only one term survives, so the result is exactly the row for "t". That's why the multiplication is replaced by a plain E[id].

The same network in PyTorch

So far we have written everything by hand. The course's real models are written in PyTorch, a library that does three things: it stores tensors, multiplies them fast (on a GPU too), and computes gradients by itself, like our Value from the last chapter, but for tensors of any size. Here is the class behind every letter network in the course, from the map at the top of this chapter to the model at the end. It comes from snapshots.py unchanged:

class MLP(nn.Module): def __init__(self, vocab, context, d_embed, hidden): super().__init__() self.context = context self.embed = nn.Embedding(vocab, d_embed) dims = [context * d_embed] + hidden self.hidden = nn.ModuleList(nn.Linear(a, b) for a, b in zip(dims, dims[1:])) self.out = nn.Linear(dims[-1], vocab) def forward(self, x): # x: (batch, context) h = self.embed(x).flatten(1) for layer in self.hidden: h = torch.tanh(layer(h)) return self.out(h)

Three details, all familiar by now:

• nn.Embedding(vocab, d_embed) is the table $E$, 97 × d_embed. Calling self.embed(x) fetches rows by number, our E[x]. During training the gradient only reaches the rows of letters that were in the batch.
• nn.Linear(a, b) is a layer of b neurons with a inputs each: a $b \times a$ weight matrix and a bias vector. Calling it computes x @ W.T + bias, a batch of dot products.
• flatten(1) glues a window's letter vectors into one: shape $(B, 3, 16)$ becomes $(B, 48)$.

The map at the top of the chapter is MLP(97, 3, 2, [128]): three letters of two numbers each and a hidden layer of 128 neurons. The last chapter's network is MLP(97, 3, 16, [256]). The model you'll meet at the end is MLP(97, 8, 24, [512, 512]): eight letters of 24 numbers and two hidden layers.

Learning in batches

How do we train such a network? Same idea as chapter 3: compute the loss, find the gradient, take a step downhill. The question is how many examples to average the loss over before each step.

We could use all of them. But our letter corpus holds tens of millions of windows, and one small step would mean running through every one. We could use a single example: steps would be quick, but each would lurch its own way, because one window says nothing about the language as a whole. The happy medium is the minibatch: take a few hundred random windows, average the loss over them and take a step. The direction isn't perfect, but it is right on average, and a batch takes a fraction of a second, thanks to matrix multiplication.

Here is the real training loop of every letter model in the course, also from snapshots.py:

def train_mlp(name, ids, vocab, context, d_embed, hidden, args, lr=3e-3, steps=None, batch=512): torch.manual_seed(0) data = windows(ids, context) n_val = len(data) // 50 val, tr = data[:n_val], data[n_val:] model = MLP(vocab, context, d_embed, hidden) opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.01) steps = steps or args.steps t0 = time.time() log = [] for step in range(steps + 1): if step % 500 == 0 or step == steps: with torch.no_grad(): idx = torch.randint(0, len(val), (8192,), generator=torch.Generator().manual_seed(0)) vb = val[idx] vl = F.cross_entropy(model(vb[:, :-1]), vb[:, -1]).item() log.append({'step': step, 'val': round(vl, 4)}) print(f'{name} step {step} val {vl:.4f} ({time.time() - t0:.0f}s)', flush=True) if step == steps: break b = tr[torch.randint(0, len(tr), (batch,))] loss = F.cross_entropy(model(b[:, :-1]), b[:, -1]) opt.zero_grad(set_to_none=True) loss.backward() opt.step() for g in opt.param_groups: g['lr'] = lr * (0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * step / steps))) return model, log

Let's read it from the top.

1. windows cuts the text into windows: a fixed number of letters of context plus the next letter, which is the right answer. The first 2% of the windows (len(data) // 50) are put aside; more on that below.
2. Each step takes 512 random windows: b[:, :-1] is the context, b[:, -1] the answers.
3. F.cross_entropy is our old friend from chapter 2: a softmax over the logits, minus the log of the right answer's probability, averaged over the batch. One function instead of three lines.
4. loss.backward() is chapter 4's backward pass: PyTorch finds the gradient for each of hundreds of thousands of parameters, rows of the embedding table included.
5. opt.step() takes the step. Instead of plain gradient descent this uses AdamW, a smarter way of choosing each parameter's step; chapter 10 takes it apart. The last two lines gently lower the learning rate towards the end.

The chapter's final model took 20,000 such steps, so it saw about 10 million windows. On a laptop with an M4 Pro that took about three minutes.

An exam on unseen text

Why put 2% of the windows aside? Because the loss on the training text can lie. A network can get very good at predicting exactly the windows it has seen, by memorising them, while understanding nothing about the language. There is only one honest test: give it text it has never seen and measure the loss there. The part set aside is the validation set; the rest is the training set.

It's easiest to see for yourself. Below is a tiny network with two numbers per letter, learning right in your browser on a slice of real stories from our corpus. On the left is its letter map; on the right, two curves: the loss on the training text and on the validation text. Start with 3,000 letters, then try 400 and the whole text.

On the whole text both curves go down together, and one by one the vowels drift away from the huddle of consonants: the map assembles itself as you watch. With 400 letters something else happens. The training loss drops far lower than on the big text, while the validation loss dips a little, then turns and climbs, until it is worse than blind guessing. The network has 1,764 parameters and fewer than 400 examples: memorising every window is easier than finding rules. This is overfitting. With 3,000 letters you can watch it set in: the validation loss falls for a while, bottoms out and slowly creeps up again.

Watch both curves during training. Training loss shows whether optimisation is working; validation loss shows whether the improvement carries over to held-out data. If the gap grows, more varied data, regularisation, a smaller model or early stopping may help. Sprout processed about 330 million tokens from a corpus of 1.43 billion. That involves fewer repeat exposures than many passes over a small dataset, but it does not rule out memorisation or repeated windows during random sampling. After choosing settings, a separate test set that played no part in model selection is useful too.

The training loss keeps falling while the validation loss rises. What's going on?

The learning rate is too small The model is memorising the training examples instead of general patterns The validation text is broken

That's overfitting. The model remembers what it has seen better and better and handles new text worse and worse. More data helps, as does a smaller model, or stopping early where the validation loss was lowest.

A wider window, a smarter model

Now we have everything we need to make the model stronger. The obvious move is to let it see more. The bigram of chapter 1 looked at one letter; last chapter's network at three. Let's take eight, give each letter 24 numbers and add a second hidden layer. We measured all these models on the same text, which none of them saw in training: 4.9 million letters.

Model	Letters seen	Parameters	Nats per letter
blind guess over 97 symbols	0	0	4.575
bigram, chapter 1	1	9,409	2.364
the map, 2 numbers a letter	3	13,603	1.874
network, chapter 4	3	39,025	1.479
network, this chapter	8	413,561	1.103

Every row is a real step forward. Even the tiny map with two numbers per letter beats the bigram by a mile: three letters of context help, and so does letting similar letters share what they learn. More numbers per letter and more neurons bring 1.479. Eight letters bring 1.103. In terms of perplexity (chapter 2), that is a fall from $e^{2.364} \approx 10.6$ "equally likely options" per letter for the bigram to $e^{1.103} \approx 3.0$.

See the difference for yourself: the three models write at the same time, from the same start, at the same temperature.

The bigram glues together word-shaped scraps. The three-letter network already writes lots of real short words. The eight-letter one strings together whole phrases from stories, like "there was a little", "One day," or "felt a", but the meaning still snaps every few words. No wonder: eight letters are a word and a half. Everything before them is invisible to the model.

Why not a window of 500 letters?

The first layer of such a network takes all the window's vectors laid end to end. For our model that is 8 × 24 = 192 numbers, and the first layer holds 192 × 512 = 98,304 weights. A 500-letter window would need 62 times as many weights in the first layer alone, most of them learning from rare cases.

Worse, every position in the window has weights of its own: a "t" in position one and the same "t" in position five go through different numbers. Whatever the network learned about "the" at the start of the window has to be learned again for every shift. That is wasteful, and there are two ways out. One is to make the pieces bigger so more text fits in the same window: that's tokens, the next chapter. The other is a mechanism that treats every position the same way and decides for itself where to look: attention, chapter 7.

Sprout right now

Sprout still reads letter by letter, but now it sees eight at once and knows which letters are alike. It writes almost real English phrases, although after a couple of words it forgets how it started. Its surprise has dropped from the bigram's 2.364 nats per letter to 1.103. In the next chapter we stop feeding it letters: the window keeps its eight slots, but each slot will hold a whole piece of a word.