Chapter 0 of 17 15 min

Meet Sprout

How do probabilities for the next piece of text become a whole answer? Meet Sprout and follow one step of generation.

In this chapter

be a language model for a minute and compare your guesses with a real one
watch a model write text one token at a time
see which parts Sprout is made of, and in which chapter we build each one

Suppose you read “Once upon a time, there was a little…” You can already imagine several ways to finish the sentence. A language model tackles a similar problem: given some text, it estimates what might come next. First it assigns probabilities to possible continuations. A small piece of text is then chosen, added to the input, and the calculation runs again. Repeating these steps produces the answer you see in a chat window.

This is the principle behind autoregressive models, including the GPT and Llama families. The rule is short; learning to make good predictions is the hard part. A useful prediction may depend on nearby words, the meaning of a sentence, or something mentioned much earlier. In this course we will work through how that prediction is made and build a small model from scratch.

Our model is called Sprout. It contains about 17 million trainable numbers, or parameters, whose values were adjusted during training on English stories and conversations. That is small for a language model: GPT-3, for example, had 175 billion parameters. Sprout can write simple stories and attempt to answer questions. Once its files have loaded, the calculations that produce its answers run in your browser.

A machine that continues text

Imagine reading a sentence that breaks off halfway: "Once upon a time there lived a king and a…". You would almost certainly say "queen". But after "In the evening I went to the…" there are plenty of options: "shop", "cinema", "park". You don't have one correct word in mind. You have a feel for the odds: some continuations fit better, others worse.

Our model first splits the text into tokens: words, parts of words, punctuation and other fragments. Each token has a number in a vocabulary. Given a sequence of these numbers, the model calculates a score for every possible next token. A function turns the scores into probabilities between 0 and 1 that add up to one. These are the model’s estimates; they do not guarantee that a continuation will make sense or be factually correct.

Be the model

The best way to understand a language model is to be one for a minute. Below are real sentence openings from the stories and dialogues Sprout learned from. Pick the word you think comes next, and Sprout will show you how it spread its own probabilities.

The buttons show several likely continuations and the word from the original example. Here we select only tokens that look like words; other candidates, such as punctuation, appear in the chart after your answer. The examples are in English, like most of Sprout’s training data.

Look at how the probability is shared between candidates. After “Once upon a time, there was a little”, “girl”, “boy” and “dog” could all fit. We did not write English grammar rules into Sprout. During training, it repeatedly predicted continuations and adjusted its parameters based on the results. That is how it learned patterns in the text, though it still makes plenty of mistakes.

At each step, the model estimates possible continuations. The token it rates most highly need not be the one the original author chose.

Token by token

To produce a whole story, we need more than one prediction. Choose the next token, append it to the input, and run the model again with that addition. This is autoregressive generation: each prediction depends on the preceding tokens, including those the model has generated itself.

Try it. Press “Roll the dice” to draw a token according to its probability: more likely candidates are chosen more often. Or tap a row to choose the continuation yourself.

Each row is a candidate for the next token, with its probability. "Most likely" always takes the top row; the dice pick at random, in proportion to the probabilities.

One step may add a whole word, part of a word, or a punctuation mark. Sprout’s vocabulary contains 8,192 tokens, including special markers. There are common words with a leading space (“ the”, “ happy”), word fragments and individual bytes, the numbers used to represent text in computer memory. This lets the tokenizer encode any text, even in a language the model barely knows. Chapter 6 explains how this vocabulary is built.

Why not always take the most likely word?

Press “Most likely” a few times. With the same starting text, model weights and calculation settings, this choice gives the same continuation each time. Sometimes it works well; sometimes it leads to a loop. Choosing the most likely token at every step does not guarantee a good story overall. Sampling allows different continuations, but it can also introduce poor choices. Chapter 11 explores how temperature, top-k and top-p control that trade-off.

What's inside

Between "text in" and "probabilities out" Sprout runs a chain of transformations. Each stage below is a chapter of this course. Tap any stage to see what happens there.

Sprout’s code contains no hand-written rules of English and no reference book of ready-made answers. It contains parameters and operations on them: matrix multiplication, addition, normalisation and nonlinear transformations. Training adjusts the parameters using examples. We hide the next token, ask the model to predict it, and measure the probability it assigned to that token. We then calculate how small changes to the parameters would affect this error and take an update step. Our pre-training run took about ten thousand such steps.

How small is it?

Is 17 million a lot or a little? Here are some famous models of recent years for comparison. Every dot in the picture is one Sprout.

This size makes the experiment possible on one computer. Sprout’s recorded pre-training run took 3 hours 41 minutes on a Mac with an M4 Pro chip; other hardware will take a different amount of time. The finished model is compact enough to run in a browser, including on a capable phone. It has many components found in larger models: transformer blocks, attention, and training through next-token prediction. Larger systems vary in architecture and training, but these foundations will help you understand them.

Try asking Sprout for a story or a simple answer. Sometimes it responds coherently; sometimes it loses the thread, repeats itself or confidently invents a fact. It was trained mainly on English, has no web search, and struggles with arithmetic. Throughout the course we will see how model size, data, training and the choice of continuation affect the result. A fluent answer is not, by itself, evidence of a reliable one.

How this course works

Every chapter adds one idea to the model. We start with something very simple, counting which letter follows which, and work our way up, step by step, to the transformer, to training on hundreds of millions of tokens and to fine-tuning for conversation. Along the way you will meet:

Widgets you can play with: drag sliders, tap, choose. Almost all of them compute for real, right on your device.
Notes on maths, on Python and "for researchers", for anyone who wants to dig deeper. Feel free to skip them.
Python code. Many cells run right on the page: Python loads into your browser the first time you press Run. Try it:

text = "once upon a time there was a little girl" counts = {} for letter in text: counts[letter] = counts.get(letter, 0) + 1 # the five most common letters for letter, n in sorted(counts.items(), key=lambda kv: -kv[1])[:5]: print(repr(letter), n)

This tiny count is the seed of the first model, which we build in the next chapter. You can edit the code: put in your own sentence and run it again.

What does a language model actually produce when you give it some text?

The model outputs a probability distribution over its whole vocabulary. Which token to take from that distribution is a separate decision, and there are many ways to make it.

The course includes short teaching examples as well as excerpts from the files used to train Sprout. The complete files and data are available in the course materials, so you can repeat the training on your own computer.

Sprout right now

We haven't written a single line of our own model yet; all we have is the goal. Here it is: the finished Sprout, fine-tuned to talk. Chat with it to see where we are heading; the menu in its bar also lets you try Sprout-2, the bigger model the course grows in its second part. By the end of the course you will have built one just like it, and you will understand what happens inside at every step. The first step comes in the next chapter: we will count which letter follows which, and that table of counts will be the first version of Sprout that actually writes.