What's next

At the start of this course, continuing text took a table: 97 rows and 97 columns of character-pair counts. Sprout's replies now come from eight transformer blocks and 17 million trained parameters. The distance between those models is a series of understandable decisions: how to represent text, use context, measure error and update weights. Chapters 14–16 then grew it into Sprout-2: 125 million parameters, 9 billion tokens of the web, code and commands, thoughts before an answer and a Python tool. In this final chapter we will compare what those decisions achieved and choose a question to investigate next.

We will first give the models the same beginning and read their continuations. Then we will check our impressions against measurements: one good story does not make a model better at every task. From there we will look at large models and experiments we can run with Sprout. Some ideas have already been tested here; for others, the course provides code and a starting point.

The whole garden

Here are all eight models of the course's first part, from the letter bigram of chapter 1 to the chat model of chapter 12, and at the end of the row Sprout-2's chat from chapter 16. The first part's models load into your browser; Sprout-2 answers from the course's server, so its 126 MB don't have to be downloaded. Give them one prompt and wake them up. They get the same random seed and the same sampling settings: temperature 0.8, top-p 0.95. The chat models receive your text as a message and answer it; all the others simply continue it.

Read the continuations from top to bottom. The bigram uses only the last character; the networks from chapters 4–5 use three or eight. Tokenization lets the same number of positions cover several words. Attention provides more flexible communication between positions within the available window; it does not make that window unlimited. Extra blocks, data and training improve the continuations. The chat model is specifically tuned to answer as an assistant, so read its output with that difference in mind.

Impressions can deceive, so here is the same road in numbers. On the same held-out text that no model has seen during training, we measured the average surprise of each model, in nats per letter. Why per letter and not per token? Because the models cut text differently: letter models make a prediction for every character, token models for every piece, and only surprise per letter can be compared fairly.

The first rung, the uniform guess, is $\ln 97 \approx 4.57$ nats: a model that knows nothing chooses among 97 symbols. Counting letter pairs (chapter 1) brings the bigram down to 2.36, and every later idea shaves off a bit more. Perplexity per letter makes it tangible: the bigram is uncertain as if it were choosing among about eleven letters, and the last rungs among fewer than two. Where the ladder shows Sprout's early checkpoints from chapter 10, notice one more thing: early in training the big model is worse than smaller models that have finished learning, and only data and steps put it ahead.

Scaling laws

Model size, data and compute budget have a systematic relationship with quality. In Scaling Laws for Neural Language Models (Kaplan et al., 2020), average loss across a wide range of experiments was approximately described by power laws. Under suitable conditions, increasing scale gave predictable improvements. This does not promise that any doubling of parameters reduces the entire loss by the same factor: data, training duration and the range in which the fit was tested matter too.

To compare models, we need a measure of cost. Training costs roughly

$$C \approx 6\,N\,D$$

operations, where $N$ is the number of parameters and $D$ the number of training tokens. Where does the 6 come from?

Now the practical question: given a budget $C$, is it better to build a bigger model and show it less data, or a smaller one and show it more? In 2022 Jordan Hoffmann and colleagues at DeepMind fitted the loss as a function of both at once:

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

Here $E$ is the loss of a perfect model (text has irreducible randomness), the second term is what the model loses by being too small, the third what it loses by having read too little. With a fixed budget, $N$ and $D$ pull against each other: a bigger model means fewer tokens. The best point sits in the valley between them. Their conclusion, the "Chinchilla rule": grow the model and the data together, at roughly 20 tokens per parameter. The 70B-parameter Chinchilla, trained on 1.4 trillion tokens, beat the 280B-parameter Gopher trained with the same compute.

Choose “Sprout”: its green dot sits near the minimum of the plotted curve. That agrees with the guide we used, 330M / 17.3M ≈ 19 tokens per parameter. But the curve comes from other experiments on other data; at our scale it is an extrapolation, not a measured optimum. Now choose GPT-3, which had fewer than two training tokens per parameter. The Chinchilla work showed that under the studied conditions this budget could be allocated more effectively to a smaller model trained on more data.

Now choose Sprout-2: 9 billion tokens for 125 million parameters, 72 tokens per parameter. Its dot sits beside the curve's minimum, on the side of the smaller model: by either fit, the minimum for its budget is near 190–215 million parameters. That was deliberate. The Chinchilla rule counts only the cost of training; a model that will answer many people, or run on a phone, is paid for again with every token it writes, so it can be worth training a smaller model for longer (Sardana and Frankle, 2023). Small open models go much further: SmolLM2-135M, close to Sprout-2 in size, read 2 trillion tokens.

import numpy as np # Chinchilla's fit (Hoffmann et al., 2022): loss from parameters N and tokens D E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28 def loss(N, D): return E + A / N ** alpha + B / D ** beta N, D = 17_309_056, 330e6 # Sprout C = 6 * N * D # training compute, in operations print(f"Sprout: C = {C:.2e}, {D / N:.0f} tokens per parameter") sizes = np.logspace(5, 13, 4000) # candidate model sizes for budget in [C, 1e18, 1e20, 1e22, 1e24]: best = sizes[np.argmin(loss(sizes, budget / (6 * sizes)))] tokens = budget / (6 * best) print(f"C = {budget:.1e}: best N = {best:9.3g}, D = {tokens:9.3g}, D/N = {tokens / best:4.0f}")

At Sprout's budget the formula recommends 17.6 million parameters, practically Sprout itself. But notice the last column: with these constants the recommended ratio drifts upwards as the budget grows. That is one reason to be careful with fitted formulas.

Your compute budget has grown 100 times. By the Chinchilla rule, what should you do?

Make the model 100 times bigger and keep the same data Make the model about 10 times bigger and give it about 10 times more data Keep the model and train it on 100 times more data

C ≈ 6·N·D, and the rule keeps D/N roughly constant. Then N and D both grow as the square root of C: 100 times the compute means about 10 times the parameters and 10 times the tokens.

From Sprout to the frontier

Compare Sprout with a large model that has a detailed technical report: Llama 3.1 405B (Meta, 2024). A specific example is more useful than a quickly outdated comparison with “the biggest model”: we know both its architecture and its training scale.

	Sprout	Llama 3.1 405B	ratio
Parameters	17.3M	405B	×23,400
Training tokens	330M	15.6T	×47,000
Compute, operations	3.4·10¹⁶	3.8·10²⁵	×1.1 billion
Context, tokens	512	128K	×256
Vocabulary	8,192	128K	×16
Hardware	one laptop, 3 h 41 min	up to 16K H100 GPUs

In compute, the gap is a factor of a billion. Yet look at what is not in the table: the architecture. Llama 3 is a decoder-only transformer with RMSNorm, rotary positions and a SwiGLU feed-forward layer, just like Sprout. It predicts the next token and minimises cross-entropy, just like Sprout. The differences begin where our course ends. Here are the main directions, briefly.

Data

At frontier scale, data is the main craft. Trillions of tokens come from the web, books, code and scientific papers, and most of the work is filtering: removing duplicates, spam and machine-generated junk, keeping the educational and the well-written, balancing languages and topics. Code and mathematics are added deliberately because they seem to help reasoning. Synthetic data, text written by other models, makes up a bigger share every year; TinyStories, one of Sprout's sources, is an early example. Chapter 9 is the same work at the scale of a laptop.

After pre-training

A base model continues text; a helpful assistant is shaped afterwards. First SFT on examples of good answers (chapter 12), then preference tuning: people compare pairs of answers and the model is pushed towards the preferred ones, via a reward model and reinforcement learning (RLHF, InstructGPT, 2022) or directly with a loss over pairs (DPO, 2023; see the note in chapter 13). This is where politeness, refusals of dangerous requests and the habit of admitting uncertainty come from.

Reasoning

Writing intermediate steps can sometimes help a model solve a problem it fails to answer directly: this was studied in the chain-of-thought work of Wei and colleagues (2022). Another route rewards a verifiable result, such as code passing tests. DeepSeek-R1 (2025) explored how such training changes solution strategies; the R1 version also used stages with prepared examples. Longer solutions spend more computation at answer time, but length alone does not guarantee correctness. A written explanation need not fully reflect the internal causes of the answer either.

Long context

Sprout has a 512-token context; our Llama 3.1 example has 128 thousand. Extending context involves both training and memory costs. Methods such as YaRN modify positional encodings; FlashAttention computes attention in blocks without retaining the full $T \times T$ matrix; grouped-query attention reduces the KV cache by sharing keys and values between heads. These address different parts of the problem. FlashAttention alone neither removes the quadratic number of full-attention pairs nor teaches a model to use distant context.

Mixture of experts

Why use every parameter on every token? In a mixture-of-experts layer the feed-forward network is split into many "experts", and a small router sends each token to one or a few of them (Shazeer et al., 2017; Switch Transformer, 2021). The model stores many parameters but uses only a fraction per token: its knowledge grows faster than its cost. Many of today's largest open models are built this way.

Seeing and hearing

Multimodal models receive more than text. One way to add vision is to divide an image into patches, process them with a visual encoder, then map its vectors into a representation compatible with the language model. CLIP (2021) is an example of learning compatible image and caption representations. Sound needs its own encoding. The components can then be joined through a shared sequence, cross-attention or another design. Turning a picture into numbers is not enough on its own: the system must also learn to use those representations together.

Tools and memory

Answering a question about current weather requires fresh information; a calculator helps with exact large products. A language model can produce a tool call, let a program execute it and receive the result back in context. Toolformer (2023) studied one version of this approach. Retrieval-augmented generation, RAG (2020), adds retrieved documents to the input. This gives the model an external source, but does not guarantee correctness: retrieval may find the wrong page, or generation may misuse what it says.

How to measure a model

Loss on held-out text measures prediction quality on that text. Question-answering needs other tests, such as benchmarks with known correct answers. In one setup, a model scores each offered continuation by summing its token $\log p$ values and chooses the largest sum. This can evaluate endings to everyday scenes, as in HellaSwag. MMLU collects questions in 57 subjects; depending on the protocol, evaluation may score answer labels instead. Prompt format and scoring rules are part of the test definition and should be stated.

We'll set Sprout's garden a small exam of this kind: twelve questions at children's-book level, four options each, the right one always among them.

The final Sprout, its chat version and Sprout-2’s base model (examined on the server) answer all twelve correctly in either mode, and the four-block gpt-4 gets eleven; it is among the smaller models that the scoring mode starts to change the result. Try both modes. Summed $\log p$ gives an advantage to short options: every extra token adds more surprise. Dividing by length removes that bias but can introduce the opposite one. The widely used lm-evaluation-harness reports both variants for exactly this reason. Here is the same effect in miniature, with a word-bigram "model" built from six short sentences:

import math from collections import Counter text = """lily was thirsty so she drank a glass of water . tom was hungry so he ate a big red apple . the cat drank a glass of milk . the dog drank water from a bowl . mom made warm soup . lily drank a glass of water and tom drank a glass of water .""" words = text.split() pairs, count = Counter(zip(words, words[1:])), Counter(words) V = len(count) def logp(context, option): """log p(option | context) under a word bigram with add-one smoothing""" prev, total = context.split()[-1], 0.0 for w in option.split(): total += math.log((pairs[(prev, w)] + 1) / (count[prev] + V)) prev = w return total question = "the dog was thirsty so it drank a glass of" options = ["water", "sand", "warm soup from a bowl", "milk"] for o in options: lp = logp(question, o) print(f"{o:>22}: sum {lp:6.2f} per word {lp / len(o.split()):6.2f}")

By the sum, the long option comes last with a huge gap; per word, it overtakes "sand". Neither way is "correct": it is a choice the author of a benchmark has to make and disclose.

Looking inside

We know exactly how Sprout computes: every multiplication is in model.py. But knowing the computation is not the same as understanding it. Why does this attention head look at the previous token? Where is "it's a story about a dog" stored? The field that asks these questions is called interpretability, and small models like Sprout are ideal for it: they are cheap to take apart, and everything fits on a laptop.

You have already used two of its tools. In chapter 7 we looked at attention maps, which show where each token looks. In chapter 8 the logit lens (nostalgebraist, 2020) let us read the residual stream after every block as if the model had to answer right there. The next steps:

• Circuits. Elhage and colleagues (2021) showed how to decompose a small transformer into paths through which information flows from token to token, and found induction heads: pairs of heads that implement the rule "if A was followed by B earlier, after A predict B again". Olsson and colleagues (2022) linked these heads to the model's ability to learn from examples in the context.
• Probes. Train a tiny classifier on the residual stream: can it tell from the vectors whether a sentence is a question, or whether a character is happy? If it can, the information is there, though not necessarily used.
• Sparse autoencoders. A neuron rarely means one thing. A sparse autoencoder unpacks the residual stream into many rarely active "features", many of which do turn out to be interpretable (Bricken et al., 2023; Templeton et al., 2024, on a production-size model).
• Interventions. The strongest test: change something inside and see what changes outside. Switch off a head, swap a vector from another text into the residual stream, and measure how the prediction changes.

Experiments to start with

The best way into research is a small experiment carried through to the end. A few rules that save weeks. Change one thing at a time. Fix the seed, and for important conclusions run two or three seeds: the difference between runs is your measure of noise. Compare runs at equal compute, not at equal steps. Start from a small budget, where a run takes minutes, and scale up only what works. Measure loss per letter when tokenizers differ. And write everything down: the settings, the curves, what you expected and what happened.

Here are experiments that fit on one laptop with this course's code. Each is a real open question at Sprout's scale.

1. Vocabulary size. Retrain the tokenizer with 4,096 and 16,384 tokens and train Sprout on each. A bigger vocabulary makes the text shorter but adds parameters to the embedding and makes rare tokens harder to learn. Compare loss per letter.≈3.5 h per run; try a quick 50M-token version first
2. Depth versus width. Take about the same number of parameters in 4 blocks of width 512, 8 of width 384 and 16 of width 256. Which learns faster, and which ends lower?three runs of ≈3.5 h
3. Your own scaling law. Train five sizes, from width 128 to 512, each at about 20 tokens per parameter, and fit a power law to the final losses. Does Sprout lie on your line?≈13 h in total, two thirds of it the biggest size
4. The data mix. Train only on TinyStories, only on SODA, and on the full mixture. Measure loss on each source separately and ask the chat model the same questions.three runs; the chat stage adds minutes
5. Ablations. Remove one part at a time: QK-norm, RoPE (without explicit rotary position encodings), weight tying, SwiGLU in favour of a plain MLP. How much does each part contribute at this scale?≈30 min each at 50M tokens
6. Muon against AdamW, all the way. We compared them over 10M tokens (chapter 10). Does Muon's lead survive a full run, or does it shrink?two full runs
7. Distillation. Train a small Sprout, 6 blocks of width 256, on a mixture of actual text targets and sprout-base’s probability distributions (Hinton et al., 2015). Does the student beat a model of the same size trained on text alone?one run plus the teacher's forward passes
8. Letters instead of tokens. Train Sprout on characters (a vocabulary of 97) with the same compute. How much worse is it per letter, and what does it fail at?≈4× more steps for the same text
9. Adapter rank and forgetting. Train the pirate adapter with ranks 1, 4, 16 and 64, and for each measure the loss on the pirate data and the surprise at plain stories, as in chapter 13's table.minutes per adapter
10. Find an induction head. Feed Sprout a random sequence of tokens repeated twice and find the heads that, on the second copy, look at the token right after the previous occurrence of the current one.no training at all

The planner estimates runtime using $6ND$ and device throughput. Its default is Sprout's measured speed on the M4 Pro. For your machine, first run a short benchmark and use the tok_s field in the train.py log. Extrapolation to another model shape is approximate: matrix sizes, context, device utilisation and evaluations also affect time. The durations in the list above are guides for our hardware, not promises for every laptop.

This distillation example gives the student two learning signals. One is ordinary cross-entropy for the actual next token. The other compares its full distribution with the teacher's distribution, softened by temperature $T$. The mix parameter sets the share of the second signal. Teacher and student use the same tokenizer and vocabulary here, so their probabilities can be compared position by position:

import torch.nn.functional as F def distill_loss(student_logits, teacher_logits, targets, T=2.0, mix=0.5): """Hinton et al. (2015): learn from the teacher's soft probabilities and the real text.""" V = student_logits.size(-1) student, teacher = student_logits.reshape(-1, V), teacher_logits.reshape(-1, V) # one row per token soft_teacher = F.softmax(teacher / T, dim=-1) log_student = F.log_softmax(student / T, dim=-1) kl = F.kl_div(log_student, soft_teacher, reduction='batchmean') * T * T # averaged over tokens ce = F.cross_entropy(student, targets.reshape(-1)) return mix * kl + (1 - mix) * ce

The $T^2$ multiplier approximately compensates for the shrinking gradients of the soft term as temperature increases; it does not guarantee identical scales for arbitrary logits. Add a frozen teacher forward pass under torch.no_grad() to the training loop and pass both sets of logits to this function. Keep the teacher in evaluation mode so training mechanisms such as dropout do not change its targets.

What to read

Papers are this field's primary sources, and most of them read better than their reputation suggests. Here are the ones behind this course, grouped by topic, with the chapter where we met each idea. You don't need to read them in order: start with whatever surprised you most.

Foundations

The transformer

Language models and scale

Training and generation

Fine-tuning and alignment

Beyond Sprout

Evaluation and interpretability

And if you'd like to watch someone build a model by hand once more, at a different pace: Andrej Karpathy's video course Neural Networks: Zero to Hero and his nanoGPT repository follow a road much like ours.

Sprout right now

We have gone from counting characters to a small conversational model. Sprout still makes mistakes and was trained mainly in English, but you can now examine its construction step by step: locate a matrix in the code, follow a calculation and test the effect of a change. Knowing what parameters do in the computation is not the same as explaining every learned number. That remains a research problem of its own.

The next time an answer appears token by token, try retracing the familiar chain: text becomes identifiers, identifiers become vectors, blocks process the context, the output layer produces scores, and decoding selects a continuation. A larger system may add retrieval, tools and further stages. For this basic chain, though, you now have a working example you can open and study.

Have one last chat with Sprout, in any of its three voices. Then open the course code, plant a seed of your own and see what grows. Thank you for growing this one with us.