What's next

Back in chapter 1 Sprout was a table: 97 rows, 97 columns, and in each cell a count of how often one letter follows another. Now it is 17 million numbers arranged in eight transformer blocks. It has read hundreds of millions of tokens of stories and conversations, and it can talk, rhyme and play a pirate. Every step of that road was something you built with your own hands: counting, surprise, descent, backpropagation, embeddings, tokens, attention, the block, the corpus, training, sampling, chat and adapters.

This last chapter has two jobs. The first is to look back: gather every version of Sprout in one place and see, with our own eyes and in honest numbers, what each idea gave. The second is to look ahead: what separates Sprout from the models everyone talks about, what researchers are working on right now, and where you can start if you want to join them.

The whole garden

Here are all eight models of the course, from the letter bigram of chapter 1 to the chat model of chapter 12. Give them one prompt and wake them up. They get the same random seed and the same sampling settings: temperature 0.8, top-p 0.95. The chat model receives your text as a message and answers it; all the others simply continue it.

Read the beds from top to bottom and you'll see the whole course in miniature. The bigram produces letters in plausible pairs and nothing more: it doesn't remember what came two letters ago. The letter networks of chapters 4 and 5 already write real words, because they see a window of three or eight letters. The token network of chapter 6 thinks in whole words and pieces of words, so the same window covers several words. With attention (chapter 7) the model can look back at the whole text rather than a fixed window, and a stack of four blocks (chapter 8) already holds a sentence together. Sprout itself, trained for three and a half hours on 330 million tokens, keeps a story going. And only the chat model understands that it is being asked something.

Impressions can deceive, so here is the same road in numbers. On the same held-out text that no model has seen during training, we measured the average surprise of each model, in nats per letter. Why per letter and not per token? Because the models cut text differently: letter models make a prediction for every character, token models for every piece, and only surprise per letter can be compared fairly.

The first rung, the uniform guess, is $\ln 97 \approx 4.57$ nats: a model that knows nothing chooses among 97 symbols. Counting letter pairs (chapter 1) brings the bigram down to 2.36, and every later idea shaves off a bit more. Perplexity per letter makes it tangible: the bigram is uncertain as if it were choosing among about eleven letters, and the last rungs among fewer than two. Where the ladder shows Sprout's early checkpoints from chapter 10, notice one more thing: early in training the big model is worse than smaller models that have finished learning, and only data and steps put it ahead.

Scaling laws

Why do bigger models write better? The answer turned out to be surprisingly regular. In 2020 Jared Kaplan and colleagues at OpenAI trained a whole family of transformers, from tiny to large, on different amounts of data, and noticed that the loss falls as a power law: double the parameters and the loss drops by the same factor, whether you start from a million or a billion. On a log-log plot that is a straight line stretching across many orders of magnitude.

To compare models, we need a measure of cost. Training costs roughly

$$C \approx 6\,N\,D$$

operations, where $N$ is the number of parameters and $D$ the number of training tokens. Where does the 6 come from?

Now the practical question: given a budget $C$, is it better to build a bigger model and show it less data, or a smaller one and show it more? In 2022 Jordan Hoffmann and colleagues at DeepMind fitted the loss as a function of both at once:

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

Here $E$ is the loss of a perfect model (text has irreducible randomness), the second term is what the model loses by being too small, the third what it loses by having read too little. With a fixed budget, $N$ and $D$ pull against each other: a bigger model means fewer tokens. The best point sits in the valley between them. Their conclusion, the "Chinchilla rule": grow the model and the data together, at roughly 20 tokens per parameter. The 70B-parameter Chinchilla, trained on 1.4 trillion tokens, beat the 280B-parameter Gopher trained with the same compute.

Choose "Sprout" and look where its green dot lands: almost exactly at the bottom of its valley. We chose Sprout's size and 330 million tokens with this rule in mind: 330M / 17.3M ≈ 19 tokens per parameter. Now choose GPT-3: a 175-billion-parameter model trained on 300 billion tokens, under two tokens per parameter. By today's standards it was far too big for its data, which is exactly what the Chinchilla paper pointed out.

import numpy as np # Chinchilla's fit (Hoffmann et al., 2022): loss from parameters N and tokens D E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28 def loss(N, D): return E + A / N ** alpha + B / D ** beta N, D = 17_309_056, 330e6 # Sprout C = 6 * N * D # training compute, in operations print(f"Sprout: C = {C:.2e}, {D / N:.0f} tokens per parameter") sizes = np.logspace(5, 13, 4000) # candidate model sizes for budget in [C, 1e18, 1e20, 1e22, 1e24]: best = sizes[np.argmin(loss(sizes, budget / (6 * sizes)))] tokens = budget / (6 * best) print(f"C = {budget:.1e}: best N = {best:9.3g}, D = {tokens:9.3g}, D/N = {tokens / best:4.0f}")

At Sprout's budget the formula recommends 17.6 million parameters, practically Sprout itself. But notice the last column: with these constants the recommended ratio drifts upwards as the budget grows. That is one reason to be careful with fitted formulas.

Your compute budget has grown 100 times. By the Chinchilla rule, what should you do?

Make the model 100 times bigger and keep the same data Make the model about 10 times bigger and give it about 10 times more data Keep the model and train it on 100 times more data

C ≈ 6·N·D, and the rule keeps D/N roughly constant. Then N and D both grow as the square root of C: 100 times the compute means about 10 times the parameters and 10 times the tokens.

From Sprout to the frontier

Let's put Sprout next to one of the largest open models whose recipe is published in detail, Llama 3 405B (Meta, 2024):

	Sprout	Llama 3 405B	ratio
Parameters	17.3M	405B	×23,400
Training tokens	330M	15.6T	×47,000
Compute, operations	3.4·10¹⁶	3.8·10²⁵	×1.1 billion
Context, tokens	512	128K	×256
Vocabulary	8,192	128K	×16
Hardware	one laptop, 3 h 41 min	up to 16K H100 GPUs

In compute, the gap is a factor of a billion. Yet look at what is not in the table: the architecture. Llama 3 is a decoder-only transformer with RMSNorm, rotary positions and a SwiGLU feed-forward layer, just like Sprout. It predicts the next token and minimises cross-entropy, just like Sprout. The differences begin where our course ends. Here are the main directions, briefly.

Data

At frontier scale, data is the main craft. Trillions of tokens come from the web, books, code and scientific papers, and most of the work is filtering: removing duplicates, spam and machine-generated junk, keeping the educational and the well-written, balancing languages and topics. Code and mathematics are added deliberately because they seem to help reasoning. Synthetic data, text written by other models, makes up a bigger share every year; TinyStories, one of Sprout's sources, is an early example. Chapter 9 is the same work at the scale of a laptop.

After pre-training

A base model continues text; a helpful assistant is shaped afterwards. First SFT on examples of good answers (chapter 12), then preference tuning: people compare pairs of answers and the model is pushed towards the preferred ones, via a reward model and reinforcement learning (RLHF, InstructGPT, 2022) or directly with a loss over pairs (DPO, 2023; see the note in chapter 13). This is where politeness, refusals of dangerous requests and the habit of admitting uncertainty come from.

Reasoning

If a model writes out intermediate steps before answering, it solves harder problems: "chain of thought" (Wei et al., 2022). Since 2024, models have been trained to do this on purpose. Reinforcement learning rewards a correct final answer on problems where correctness can be checked automatically, like maths with a known answer or code with tests, and the model learns on its own to think longer, check its work and backtrack (DeepSeek-R1, 2025). Such a model spends compute not only in training but also on every answer.

Long context

Sprout sees 512 tokens; modern models see hundreds of thousands. That requires three things: positions that behave sensibly beyond the training length (stretching RoPE's frequencies, as in YaRN, 2023), attention that doesn't store a $T \times T$ matrix in memory (FlashAttention, 2022), and a key–value cache that fits in memory (fewer key and value heads, as in grouped-query attention, 2023; see chapter 11).

Mixture of experts

Why use every parameter on every token? In a mixture-of-experts layer the feed-forward network is split into many "experts", and a small router sends each token to one or a few of them (Shazeer et al., 2017; Switch Transformer, 2021). The model stores many parameters but uses only a fraction per token: its knowledge grows faster than its cost. Many of today's largest open models are built this way.

Seeing and hearing

Pictures and sound become tokens too. An image is cut into patches, each patch is turned into a vector by a separate encoder (CLIP, 2021, taught such encoders to match pictures with their captions), and those vectors enter the same transformer next to text tokens. For the transformer, a photo is just another stretch of the sequence.

Tools and memory

A model cannot know today's weather or reliably multiply twelve-digit numbers, but it can learn to call something that can. It is trained to emit special tokens that mean "call the calculator with these arguments" or "search the documents" (Toolformer, 2023), and the result is written back into the context. Retrieval (RAG, 2020) solves the problem of knowledge we met in chapter 13: rather than stuffing facts into the weights, show the model the right page at the right moment.

How to measure a model

Loss on held-out text, as on our ladder, is the most honest measure, but it says little about whether a model can answer questions. So researchers use benchmarks: sets of tasks with known answers. The simplest kind needs no text generation at all. A question has several options, and we check which option the model finds most likely, by summing $\log p$ over the option's tokens: the surprise from chapter 2 with its sign flipped. HellaSwag (2019) is built this way, with endings to everyday situations, and so is the multiple-choice part of MMLU (2020), with questions from 57 school and university subjects.

We'll set Sprout's garden a small exam of this kind: twelve questions at children's-book level, four options each, the right one always among them.

The final Sprout and its chat version answer all twelve correctly in either mode, and the four-block gpt-4 gets eleven; it is among the smaller models that the scoring mode starts to change the result. Try both modes. Summed $\log p$ gives an advantage to short options: every extra token adds more surprise. Dividing by length removes that bias but can introduce the opposite one. The widely used lm-evaluation-harness reports both variants for exactly this reason. Here is the same effect in miniature, with a word-bigram "model" built from six short sentences:

import math from collections import Counter text = """lily was thirsty so she drank a glass of water . tom was hungry so he ate a big red apple . the cat drank a glass of milk . the dog drank water from a bowl . mom made warm soup . lily drank a glass of water and tom drank a glass of water .""" words = text.split() pairs, count = Counter(zip(words, words[1:])), Counter(words) V = len(count) def logp(context, option): """log p(option | context) under a word bigram with add-one smoothing""" prev, total = context.split()[-1], 0.0 for w in option.split(): total += math.log((pairs[(prev, w)] + 1) / (count[prev] + V)) prev = w return total question = "the dog was thirsty so it drank a glass of" options = ["water", "sand", "warm soup from a bowl", "milk"] for o in options: lp = logp(question, o) print(f"{o:>22}: sum {lp:6.2f} per word {lp / len(o.split()):6.2f}")

By the sum, the long option comes last with a huge gap; per word, it overtakes "sand". Neither way is "correct": it is a choice the author of a benchmark has to make and disclose.

Looking inside

We know exactly how Sprout computes: every multiplication is in model.py. But knowing the computation is not the same as understanding it. Why does this attention head look at the previous token? Where is "it's a story about a dog" stored? The field that asks these questions is called interpretability, and small models like Sprout are ideal for it: they are cheap to take apart, and everything fits on a laptop.

You have already used two of its tools. In chapter 7 we looked at attention maps, which show where each token looks. In chapter 8 the logit lens (nostalgebraist, 2020) let us read the residual stream after every block as if the model had to answer right there. The next steps:

• Circuits. Elhage and colleagues (2021) showed how to decompose a small transformer into paths through which information flows from token to token, and found induction heads: pairs of heads that implement the rule "if A was followed by B earlier, after A predict B again". Olsson and colleagues (2022) linked these heads to the model's ability to learn from examples in the context.
• Probes. Train a tiny classifier on the residual stream: can it tell from the vectors whether a sentence is a question, or whether a character is happy? If it can, the information is there, though not necessarily used.
• Sparse autoencoders. A neuron rarely means one thing. A sparse autoencoder unpacks the residual stream into many rarely active "features", many of which do turn out to be interpretable (Bricken et al., 2023; Templeton et al., 2024, on a production-size model).
• Interventions. The strongest test: change something inside and see what changes outside. Switch off a head, swap a vector from another text into the residual stream, and measure how the prediction changes.

Experiments to start with

The best way into research is a small experiment carried through to the end. A few rules that save weeks. Change one thing at a time. Fix the seed, and for important conclusions run two or three seeds: the difference between runs is your measure of noise. Compare runs at equal compute, not at equal steps. Start from a small budget, where a run takes minutes, and scale up only what works. Measure loss per letter when tokenizers differ. And write everything down: the settings, the curves, what you expected and what happened.

Here are experiments that fit on one laptop with this course's code. Each is a real open question at Sprout's scale.

1. Vocabulary size. Retrain the tokenizer with 4,096 and 16,384 tokens and train Sprout on each. A bigger vocabulary makes the text shorter but adds parameters to the embedding and makes rare tokens harder to learn. Compare loss per letter.≈3.5 h per run; try a quick 50M-token version first
2. Depth versus width. Take about the same number of parameters in 4 blocks of width 512, 8 of width 384 and 16 of width 256. Which learns faster, and which ends lower?three runs of ≈3.5 h
3. Your own scaling law. Train five sizes, from width 128 to 512, each at about 20 tokens per parameter, and fit a power law to the final losses. Does Sprout lie on your line?≈13 h in total, two thirds of it the biggest size
4. The data mix. Train only on TinyStories, only on SODA, and on the full mixture. Measure loss on each source separately and ask the chat model the same questions.three runs; the chat stage adds minutes
5. Ablations. Remove one part at a time: QK-norm, RoPE (with no position information at all), weight tying, SwiGLU in favour of a plain MLP. How much does each part contribute at this scale?≈30 min each at 50M tokens
6. Muon against AdamW, all the way. We compared them over 10M tokens (chapter 10). Does Muon's lead survive a full run, or does it shrink?two full runs
7. Distillation. Train a small Sprout, 6 blocks of width 256, not on the next token of the text but on sprout-base's whole probability distribution (Hinton et al., 2015). Does the student beat a model of the same size trained on text alone?one run plus the teacher's forward passes
8. Letters instead of tokens. Train Sprout on characters (a vocabulary of 97) with the same compute. How much worse is it per letter, and what does it fail at?≈4× more steps for the same text
9. Adapter rank and forgetting. Train the pirate adapter with ranks 1, 4, 16 and 64, and for each measure the loss on the pirate data and the surprise at plain stories, as in chapter 13's table.minutes per adapter
10. Find an induction head. Feed Sprout a random sequence of tokens repeated twice and find the heads that, on the second copy, look at the token right after the previous occurrence of the current one.no training at all

How long will your experiment take? The planner estimates it from the 6·N·D formula and the speed of your machine. By default it uses Sprout's own speed on the M4 Pro; train.py prints your own speed as tok_s every hundred steps.

For the distillation experiment only the loss changes. Instead of cross-entropy with the single correct token, the student is compared with the teacher's whole distribution, softened by a temperature $T$ (chapter 11):

import torch.nn.functional as F def distill_loss(student_logits, teacher_logits, targets, T=2.0, mix=0.5): """Hinton et al. (2015): learn from the teacher's soft probabilities and the real text.""" V = student_logits.size(-1) student, teacher = student_logits.reshape(-1, V), teacher_logits.reshape(-1, V) # one row per token soft_teacher = F.softmax(teacher / T, dim=-1) log_student = F.log_softmax(student / T, dim=-1) kl = F.kl_div(log_student, soft_teacher, reduction='batchmean') * T * T # averaged over tokens ce = F.cross_entropy(student, targets.reshape(-1)) return mix * kl + (1 - mix) * ce

The multiplier $T^2$ keeps the gradients of the soft part at the same scale when you change the temperature. Everything else is train.py unchanged, plus one teacher forward pass under torch.no_grad() per batch.

What to read

Papers are this field's primary sources, and most of them read better than their reputation suggests. Here are the ones behind this course, grouped by topic, with the chapter where we met each idea. You don't need to read them in order: start with whatever surprised you most.

Foundations

The transformer

Language models and scale

Training and generation

Fine-tuning and alignment

Beyond Sprout

Evaluation and interpretability

And if you'd like to watch someone build a model by hand once more, at a different pace: Andrej Karpathy's video course Neural Networks: Zero to Hero and his nanoGPT repository follow a road much like ours.

Sprout right now

That's the whole road, from counting letters to a model that talks back. Sprout is small, it makes mistakes and it speaks only English, but you know every number in it: where it came from, what it does and how to change it.

Remember how the course began? Whatever chatbot you talk to, the other side does just one thing, over and over: it looks at the text and guesses what comes next. Now you know what goes into that guess: letter counts that became probabilities, surprise that became a loss, a patient walk downhill, attention that looks back over the whole text, and twenty thousand conversations that taught it to take turns. The giants are built from the same parts. The next time one of them answers you, you will know what is happening at the other end.

Have one last chat with Sprout, in any of its three voices. Then open the course code, plant a seed of your own and see what grows. Thank you for growing this one with us.