Your own LLM from scratch
Glossary
Every term in the course, briefly and plainly. Each comes with its Russian name and the chapter where it first appears.
- AdamW AdamW
- A popular optimiser: momentum plus a separate step size for every parameter. chapter 10 →
- Attention Внимание
- The mechanism by which every token gathers information from earlier ones, deciding whom to look at. chapter 7 →
- Attention head Голова внимания
- One independent attention. Several heads look at the text in different ways at once. chapter 7 →
- Autoregression Авторегрессия
- A way to write text: pick the next token, append it, and ask the model again about the one after that. chapter 0 →
- Backpropagation Обратное распространение
- A way to get all gradients at once: walk the computation graph backwards from the loss, applying the chain rule. chapter 4 →
- Base model Базовая модель
- A model after pre-training: it continues text but does not answer like a conversation partner. chapter 12 →
- Batch Батч
- The group of examples behind one training step. Sprout’s is 32,768 tokens. chapter 10 →
- Bigram Биграмма
- The simplest model: the probability of the next symbol depends only on the previous one. chapter 1 →
- BPE BPE
- Byte Pair Encoding: build a vocabulary by repeatedly merging the most frequent pair of neighbouring tokens. chapter 6 →
- Causal mask Каузальная маска
- A ban on looking ahead: a token sees only itself and what came before. chapter 7 →
- Chain rule Цепное правило
- The derivative of a composition is the product of the derivatives of its parts. chapter 4 →
- Chat template Шаблон чата
- How a conversation is written as tokens with role marks: <|user|>, <|assistant|>, <|end|>. chapter 12 →
- Conditional probability Условная вероятность
- The probability of an event given that another one has happened: p(b | a). chapter 1 →
- Corpus Корпус
- All the text a model learns from. Sprout’s has 1.43 billion tokens. chapter 9 →
- Cross-entropy Кросс-энтропия
- The model's average surprise on real text. The main loss of language models. chapter 2 →
- Deduplication Дедупликация
- Removing repeats from the corpus so the model does not cram the same text. chapter 9 →
- Derivative Производная
- How fast a function changes: how much it grows if you nudge its input. chapter 3 →
- Dot product Скалярное произведение
- The sum of the products of two vectors’ coordinates: a measure of how alike they are. chapter 5 →
- Embedding Эмбеддинг
- A learned vector of numbers for every token. Similar tokens end up with nearby vectors. chapter 5 →
- Entropy Энтропия
- The average surprise of a distribution itself: how uncertain it is. chapter 2 →
- Gradient Градиент
- The vector of derivatives with respect to all parameters. It points where the loss grows fastest. chapter 3 →
- Gradient descent Градиентный спуск
- Step by step, move the parameters against the gradient so the loss goes down. chapter 3 →
- KV cache KV-кэш
- Saved keys and values of tokens already read, so they are not recomputed at every step. chapter 11 →
- Language model Языковая модель
- A program that, given the beginning of a text, estimates the probability of every possible continuation. chapter 0 →
- Learning rate Скорость обучения
- The size of a gradient step. Too small and training takes forever, too big and it blows up. chapter 3 →
- Likelihood Правдоподобие
- The probability a model assigns to a real text. The higher, the better it predicts it. chapter 2 →
- Logits Логиты
- The model's raw scores for every token before softmax: any numbers, not probabilities. chapter 3 →
- LoRA LoRA
- Fine-tuning through small low-rank additions to the weights: 2–3 % of the parameters change. chapter 13 →
- Loss Функция потерь
- A single number saying how wrong the model is. Training makes it smaller. chapter 3 →
- Matrix rank Ранг матрицы
- How many independent directions a matrix holds. A rank-r matrix is a sum of r simple layers. chapter 13 →
- Muon Muon
- An optimiser that orthogonalises the update of every matrix. Sprout learned faster with it than with AdamW. chapter 10 →
- Nat, bit Нат, бит
- Units of information: nats with the natural log, bits with log base 2. 1 nat ≈ 1.44 bits. chapter 2 →
- Neuron Нейрон
- A weighted sum of inputs followed by a non-linearity (tanh, ReLU, …). chapter 4 →
- Overfitting Переобучение
- The model memorises its training examples instead of general patterns and does worse on new text. chapter 5 →
- Parameter Параметр
- One number inside the model that training adjusts. Sprout has 17.3 million of them. chapter 0 →
- Perplexity Перплексия
- e to the power of the cross-entropy: how many equally likely options the model seems to choose from. chapter 2 →
- Quantisation Квантование
- Storing the weights with fewer bits (8 in Sprout’s browser version) at almost no loss of quality. chapter 11 →
- Query, key, value Запрос, ключ, значение
- The three vectors of attention: the query asks, the key says what a token is about, the value is what it passes on. chapter 7 →
- Residual stream Остаточный поток
- The vector that flows through all blocks; every block adds its own correction to it. chapter 8 →
- RLHF, DPO RLHF, DPO
- Training on human preferences: which of two answers is better. chapter 14 →
- RMSNorm RMSNorm
- Normalisation: divide a vector by its root-mean-square so the numbers stay in range. chapter 8 →
- RoPE RoPE
- Rotary position embeddings: queries and keys are rotated by an angle that depends on the position. chapter 8 →
- Sampling Выборка (сэмплирование)
- Choosing a value at random, weighted by the probabilities, like rolling a loaded die. chapter 1 →
- Scaling laws Законы масштабирования
- Empirical formulas for how the loss falls as the model, the data and the compute grow. chapter 10 →
- SFT SFT
- Supervised fine-tuning: training on examples of good answers, with the loss only on the assistant's words. chapter 12 →
- Softmax Softmax
- Turns logits into probabilities: exponentiate each one and divide by the sum. chapter 3 →
- Surprise (information) Удивление (информация)
- Minus the log of the probability: −log p. Unlikely events are more surprising. chapter 2 →
- SwiGLU SwiGLU
- A gated feed-forward layer: silu(xW₁) ⊙ xW₃. chapter 8 →
- Temperature Температура
- The divisor of the logits before softmax: lower makes the model stick to its favourites, higher makes it more varied. chapter 11 →
- Token Токен
- A piece of text the model sees as one unit: a word, part of a word, a symbol. chapter 6 →
- Top-k, top-p Top-k, top-p
- Cut off the unlikely tail: keep the best k tokens, or as many as add up to probability p. chapter 11 →
- Transformer Трансформер
- An architecture of "attention + feed-forward" blocks with residual connections. The basis of every modern LLM. chapter 8 →