Chapter 0 of 14 15 min

Meet Sprout

A language model is a machine that guesses the next word. Let's see how it does that, and agree on what we are going to build.

In this chapter

be a language model for a minute and compare your guesses with a real one
watch a model write text one token at a time
see which parts Sprout is made of, and in which chapter we build each one

Every time you send a message to a chatbot, the other side does just one thing, over and over: the model looks at the text and guesses what comes next. Not the whole answer, not a thought, not a plan. Just the next small piece of text. Then another. And another.

It sounds far too simple to add up to a conversation partner. Yet this is exactly how every large language model works, from GPT to Llama. In this course we will build one from scratch, with our own hands, and watch "guess the next word" grow into coherent speech.

Our model is called Sprout. It is tiny, with 17 million numbers against the hundreds of billions inside the big models, but it is real. It was trained on English stories and dialogues, it can keep up a simple conversation, and it runs right in your browser, with no server behind it.

A machine that continues text

Imagine reading a sentence that breaks off halfway: "Once upon a time there lived a king and a…". You would almost certainly say "queen". But after "In the evening I went to the…" there are plenty of options: "shop", "cinema", "park". You don't have one correct word in mind. You have a feel for the odds: some continuations fit better, others worse.

A language model does the same, only explicitly and with numbers. Text goes in. Out comes a list of every piece of text the model knows, each with a number between 0 and 1 that says how likely it is to come next. Together, all those numbers add up to one.

Be the model

The best way to understand a language model is to be one for a minute. Below are real sentence openings from the stories and dialogues Sprout learned from. Pick the word you think comes next, and Sprout will show you how it spread its own probabilities.

The buttons are the five words Sprout finds likeliest (plus the word that really came next in the corpus, if it is not among them).

Notice two things. First, Sprout is almost never 100% sure. Even after "Once upon a time, there was a little" it leaves room for "girl", "boy" and "dog". Second, its guesses are sensible. Nobody taught it grammar or how fairy tales go; it simply read hundreds of millions of words and remembered what usually follows what.

A language model does not know "the right answer". It knows a distribution: how likely each piece of text is to come next.

Word by word

If a model can only guess the next word, how do we get a whole story? Simple: pick a word, add it to the text and ask the model again, this time about the word after that. This loop is called autoregression: the model keeps feeding on its own output.

Try it. Press "Roll the dice" and Sprout will pick a word at random, weighted by the probabilities. Or tap a row to choose the continuation yourself.

Each row is a candidate for the next token, with its probability. "Most likely" always takes the top row; the dice pick at random, in proportion to the probabilities.

You probably noticed that a step sometimes adds not a whole word but part of one, or a punctuation mark. That's because the model works not with words but with tokens, frequent pieces of text. Sprout knows 8,192 of them: whole common words (" the", " happy"), parts of rare words, punctuation, digits and a few special marks. Chapter 6 shows how such a vocabulary is made.

Why not always take the most likely word?

Try pressing only "Most likely": you get the most predictable story there is, about a little girl named Lily and her big red ball, and exactly the same one every time. The smaller models of the early chapters fare worse: always taking the top word, they soon start going round in circles. People don't always say the most predictable thing either. A little randomness brings text to life; too much makes it fall apart. Chapter 11 shows how to strike the balance with temperature, top-k and top-p.

What's inside

Between "text in" and "probabilities out" Sprout runs a chain of transformations. Each stage below is a chapter of this course. Tap any stage to see what happens there.

The most surprising thing about this picture is what is missing from it. There are no grammar rules, no dictionary of meanings, no list of facts. There are only 17 million numbers, called parameters, and simple arithmetic: multiply, add, take an exponent. Everything Sprout "knows" lives in the values of those numbers, and training is how those values are found. We show the model some text, measure how wrong it was, and nudge every number slightly towards the right answer. Then again, ten thousand times over.

How small is it?

Is 17 million a lot or a little? Here are some famous models of recent years for comparison. Every dot in the picture is one Sprout.

We chose this size on purpose. A 17-million-parameter model can be trained overnight on an ordinary laptop (Sprout trained for 3 hours 41 minutes on a Mac with an M4 Pro chip), and it is light enough to run in a phone's browser. Yet it works on exactly the same principles as the giants: the same transformer, the same training tricks, the same fine-tuning for conversation. The difference is scale, not design.

So let's be honest about what it can and can't do. Sprout writes simple stories, makes small talk, tells jokes at the level of a children's book and finds kind words when you are sad. It is bad at arithmetic (ask it "What is 7 plus 5?"), mixes up facts (it may well tell you that the capital of France is Rome), knows nothing about the news and understands only English. All of this follows directly from its size and its data, and by the end of the course you will know exactly why.

How this course works

Every chapter adds one idea to the model. We start with something very simple, counting which letter follows which, and work our way up, step by step, to the transformer, to training on hundreds of millions of tokens and to fine-tuning for conversation. Along the way you will meet:

Widgets you can play with: drag sliders, tap, choose. Almost all of them compute for real, right on your device.
Notes on maths, on Python and "for researchers", for anyone who wants to dig deeper. Feel free to skip them.
Python code. Many cells run right on the page: Python loads into your browser the first time you press Run. Try it:

text = "once upon a time there was a little girl" counts = {} for letter in text: counts[letter] = counts.get(letter, 0) + 1 # the five most common letters for letter, n in sorted(counts.items(), key=lambda kv: -kv[1])[:5]: print(repr(letter), n)

This tiny count is the seed of the first model, which we build in the next chapter. You can edit the code: put in your own sentence and run it again.

What does a language model actually produce when you give it some text?

The model outputs a probability distribution over its whole vocabulary. Which token to take from that distribution is a separate decision, and there are many ways to make it.

All the code in this course is real: these are exactly the files Sprout was trained with. You can download them from the course page and repeat every step yourself.

Sprout right now

We haven't written a single line of our own model yet; all we have is the goal. Here it is: the finished Sprout, fine-tuned to talk. Chat with it to see where we are heading. By the end of the course you will have built one just like it, and you will understand what happens inside at every step. The first step comes in the next chapter: we will count which letter follows which, and that table of counts will be the first version of Sprout that actually writes.