← Sprout Your own LLM from scratch Glossary Code RU

Interactive course · 15 chapters

Let's grow our own language model

Meet Sprout, a tiny LLM with 17 million parameters. It was trained from scratch on English stories and conversations, and it lives right in your browser. Over fifteen chapters we will build the same model with our own hands: from counting letters to chatting, with the maths, the code and live widgets at every step.

Start with chapter one → 💬 Talk to Sprout
  • 17Mparameters
  • 8,192tokens in the vocabulary
  • ~330Mtraining tokens
  • 0servers: it all runs in your browser

Talk to Sprout

Sprout only understands English and sometimes gets things wrong: it is very small. Everything runs on your device.

How Sprout grew

Every chapter adds one idea to the model, and the loss goes down. Here are all versions of Sprout, from picking letters at random to the full model, on the same text.

The path

Every chapter adds one new idea to the model. At the end of each one you meet Sprout as far as we have grown it.

1Basics: probability and learning

  1. 0 Meet Sprout A language model is a machine that guesses the next word. Let's see how it does that, and agree on what we are going to build. ✓ done
  2. 1 Counting letters The simplest language model there is: a table of which letter follows which, counted over 41 million letters of stories. Let's build it and let it write. ✓ done
  3. 2 Measuring surprise How good is a language model? We show it real text and measure its surprise, and on the way meet logarithms, bits and nats, entropy, and the one number all of training is about: the loss. ✓ done
  4. 3 Walking downhill Counting is not the only road to good probabilities: you can start from any numbers at all and walk, step by step, down to where the error is smaller. That is how every neural network learns, from a bigram to Sprout. ✓ done

2Inside a neural network

  1. 4 How blame flows backwards To train a model by descent you need to know how much each of its numbers is to blame for the error. Backpropagation finds that for every number at once, and by the end of this chapter you will have written it yourself. ✓ done
  2. 5 Letters as points The network from the last chapter kept a short list of numbers for every letter. Let's see what those numbers are, why similar letters end up side by side, and how to train such a network on millions of examples. ✓ done
  3. 6 Tokens A letter is too small a piece and a word too big. Let's see how Sprout cuts up text, where its 8,192 tokens came from, and why it sees Russian as a scatter of bytes. ✓ done

3The transformer

  1. 7 Attention How a token finds the words it needs among everything that came before. Queries, keys and values: the transformer's central idea, taken apart piece by piece. ✓ done
  2. 8 The transformer Putting all of Sprout together: a block of attention plus a small network, the residual stream, normalisation, positions as rotations, and the whole of model.py, line by line. ✓ done

4Training for real

  1. 9 The corpus A model is what it has read. Here is where Sprout's 1.4 billion tokens came from, how we cleaned them, and how much text a model of this size actually needs. ✓ done
  2. 10 Training Ten thousand steps, 330 million tokens, three and a half hours on one Mac: the training loop line by line, the optimizer that gave us a head start, the schedule, and a time-lapse of Sprout learning to write. ✓ done
  3. 11 Choosing the next word Sprout hands us 8,192 probabilities, and which word actually gets written is up to us. Meet temperature, top-k, top-p and the repetition penalty, and see why the model still runs fast on a phone. ✓ done

5Conversation and fine-tuning

  1. 12 Teaching it to talk A base model continues text; it doesn't answer. Four special tokens, twenty thousand conversations and a loss that ignores everything the user says turn a storyteller into a conversation partner. ✓ done
  2. 13 Your own fine-tune How to teach Sprout a new way of speaking without touching its 17 million numbers: two thin matrices beside every layer and a few hundred examples. ✓ done
  3. 14 What's next Every version of Sprout in one garden, the laws that separate it from the giants, and the experiments you can start tomorrow. ✓ done

What you need to know

  • School maths: fractions, powers, graphs. We explain everything else on the way.
  • A little Python, or the patience to read code line by line with us.
  • A computer: a browser is enough for the first chapters; for training the full model, any laptop with a GPU or a Mac with an M chip.
📖 Glossary 🐍 Sprout's code