← Sprout Your own LLM from scratch Glossary Code RU

Free interactive course · 18 chapters

Sprouta language model grown from scratch

Meet Sprout-2: 125 million parameters, code and the command line, thinking before it answers, and Python. Talk to it, then learn from the course how it works and how to grow a model like it yourself.

Start with chapter one → How Sprout-2 was grown
  • 125Mparameters in Sprout-2
  • 9Btokens read
  • 30transformer layers
  • 0sign-ups or fees

Interactive course · 18 chapters

Let's grow our own language model

Sprout is a small language model we grow from scratch. Over fourteen chapters we will work from a table of frequencies to a transformer with 17 million parameters, using explanations, Python code and browser experiments. Then we grow it into Sprout-2: 125 million parameters, code and the command line, thinking before it answers and running Python.

Start with chapter one → The path ↓

How Sprout grew

Compare the course models on the same held-out text, from random character guesses to the trained transformer. The chart shows how their prediction accuracy changes.

The path

Every chapter adds one new idea to the model. At the end of each one you meet Sprout as far as we have grown it.

1Basics: probability and learning

  1. Chapter 0 Meet Sprout How do probabilities for the next piece of text become a whole answer? Meet Sprout and follow one step of generation. ✓ done
  2. Chapter 1 Counting letters Count neighbouring characters in English stories and conversations. Their frequencies will give us our first model that can continue a piece of text. ✓ done
  3. Chapter 2 Measuring surprise How can we compare two text predictors? Build a numerical score and find out what loss, bits and perplexity tell us. ✓ done
  4. Chapter 3 Walking downhill Find the probabilities again, this time by updating logits in small steps. Along the way, learn what derivatives, gradients and learning rates do. ✓ done

2Inside a neural network

  1. Chapter 4 Following the gradient backwards How do we find gradients through a long chain of operations? Trace a backward pass and write a small automatic differentiation system. ✓ done
  2. Chapter 5 Letters as points Replace character identifiers with trainable vectors. Explore their map, learn the arithmetic behind them, and follow training through two loss curves. ✓ done
  3. Chapter 6 Tokens Eight tokens can hold more text than eight letters. Explore BPE, Sprout’s byte vocabulary, and why different languages use different numbers of positions. ✓ done

3The transformer

  1. Chapter 7 Attention Let the model mix information from different parts of its context. Work through queries, keys and values, then learn what an attention map can actually show. ✓ done
  2. Chapter 8 The transformer Follow a vector from its embedding to the next-token prediction. Assemble Sprout’s block, work through normalisation and positions, then count every parameter. ✓ done

4Training for real

  1. Chapter 9 The corpus Sprout needs a library. We will look at its sources, prepare the texts for training, and see why the size of a corpus is not the amount a model actually reads. ✓ done
  2. Chapter 10 Training How do 10,070 steps turn random weights into a model that writes stories? Follow the training loop, compare optimizers, and examine samples from the actual run. ✓ done
  3. Chapter 11 Choosing the next word The model supplies continuation probabilities; the program must choose one token. Explore temperature, top-k and top-p, then see how caching and quantisation speed up generation. ✓ done

5Conversation and fine-tuning

  1. Chapter 12 Teaching it to talk How does a text continuation become a reply to a person? Prepare conversations, examine role markers and the loss mask, then see what fine-tuning changes. ✓ done
  2. Chapter 13 Your own fine-tune Freeze the original weights and train small corrections: one for a pirate voice and one for verse. Use LoRA to explore matrix rank, memory savings and how to check for lost abilities. ✓ done

6Sprout-2: more data, a bigger model, new skills

  1. Chapter 14 More data Sprout-2's corpus is ten and a half billion tokens of web pages, code, questions and answers, and maths. Where such a pile comes from, what its licences allow, and what must happen to it before the first training step. ✓ done
  2. Chapter 15 A bigger model Sprout-2 has 125 million parameters in thirty narrow blocks. Where those parameters sit, what a model of this size needs that a small one got away without, how we tried it all on small copies first, and how nine billion tokens of training went. ✓ done
  3. Chapter 16 Thinking and a Python tool The base model continues text. A chat format with more roles, a quarter of a million conversations and four rounds of fixes turn it into a chat that thinks before it answers and hands arithmetic to a Python program that really runs. ✓ done

7Onward

  1. Chapter 17 What's next Compare the course models on the same examples, examine the limits of scaling laws, and plan experiments of your own, from tokenization to fine-tuning. ✓ done

What you need to know

  • School maths: fractions, powers, graphs. We explain everything else on the way.
  • A little Python, or the patience to read code line by line with us.
  • A browser for reading and experiments. For your own training runs, a computer with Python and PyTorch; a supported GPU or an Apple silicon Mac makes training considerably faster.
📖 Glossary 🐍 Sprout's code