Your own LLM from scratch
Glossary
Every term in the course, briefly and plainly. Each comes with its Russian name and the chapter where it first appears.
- AdamW AdamW
- An optimizer that scales updates using running averages of gradients and squared gradients, with weight decay applied separately. chapter 10 →
- Attention Внимание
- A weighted mixture of value vectors. In Sprout’s causal attention, each position uses itself and available earlier positions. chapter 7 →
- Attention head Голова внимания
- One set of query, key and value projections. Multiple heads compute attention in parallel before their outputs are combined. chapter 7 →
- Autoregression Авторегрессия
- A way to write text: pick the next token, append it, and ask the model again about the one after that. chapter 0 →
- Backpropagation Обратное распространение
- A way to get all gradients at once: walk the computation graph backwards from the loss, applying the chain rule. chapter 4 →
- Base model Базовая модель
- A model after pre-training and before further tuning. It may continue text and answer some questions, but has not yet been adapted to the assistant role. chapter 12 →
- Batch Батч
- The group of examples behind one training step. Sprout’s is 32,768 tokens. chapter 10 →
- Bigram Биграмма
- A pair of adjacent units: characters, words or tokens. Our character bigram model predicts the next character using only the previous one. chapter 1 →
- BPE BPE
- Byte Pair Encoding: build a vocabulary by repeatedly merging the most frequent pair of neighbouring tokens. chapter 6 →
- Causal mask Каузальная маска
- A ban on looking ahead: a token sees only itself and what came before. chapter 7 →
- Chain rule Цепное правило
- The derivative of a composition is the product of the derivatives of its parts. chapter 4 →
- Chat template Шаблон чата
- How a conversation is written as tokens with role marks: <|user|>, <|assistant|>, <|end|>. chapter 12 →
- Conditional probability Условная вероятность
- The probability of an event given that another one has happened: p(b | a). chapter 1 →
- Cooldown Спуск скорости обучения
- The last part of training, where the learning rate falls smoothly to zero. Sprout-2 kept a checkpoint from just before it, so training can be continued later. chapter 15 →
- Corpus Корпус
- A collection of texts for training and evaluation. Sprout’s prepared data has 1.43B tokens; its main run sampled about 330M, including possible repeats. chapter 9 →
- Cross-entropy Кросс-энтропия
- The average of −log q for events drawn from a distribution p, where q is the model prediction. Here we estimate it from the actual text continuations. chapter 2 →
- Data mix Смесь данных
- The share of each source in training: how many tokens the model sees from the web, code, Q&A and so on. Sprout-2 switched to a different mix for the cooldown. chapter 14 →
- Decontamination Деконтаминация
- Removing training documents that overlap with benchmark questions (for Sprout-2, 13 words in a row), so that a test measures skill, not memory. chapter 14 →
- Deduplication Дедупликация
- Removing repeats from the corpus so the model does not cram the same text. chapter 9 →
- Derivative Производная
- How fast a function changes: how much it grows if you nudge its input. chapter 3 →
- Document mask Маска документов
- A training row holds several documents back to back; the mask stops a token from attending to tokens of another document. chapter 15 →
- Dot product Скалярное произведение
- The sum of coordinate-wise products. It depends on vector directions and lengths, and is often used to calculate similarity scores. chapter 5 →
- Embedding Эмбеддинг
- A vector representation of an object. In Sprout each token has a trainable table row; geometric relationships between vectors depend on training. chapter 5 →
- Entropy Энтропия
- The average surprise of a distribution itself: how uncertain it is. chapter 2 →
- Fill-in-the-middle (FIM) Заполнение середины (FIM)
- A code file is cut in three and reordered with special tokens: the beginning, the end, then the middle. The model learns to write code between existing lines. chapter 14 →
- Gradient Градиент
- The vector of derivatives with respect to all parameters. It points where the loss grows fastest. chapter 3 →
- Gradient descent Градиентный спуск
- Step by step, move the parameters against the gradient so the loss goes down. chapter 3 →
- Grouped-query attention (GQA) Групповое внимание (GQA)
- Several query heads share one key/value head. Sprout-2 has 9 query heads and 3 key/value heads, so its KV cache is three times smaller. chapter 15 →
- KV cache KV-кэш
- Saved keys and values of tokens already read, so they are not recomputed at every step. chapter 11 →
- Language model Языковая модель
- A program that, given the beginning of a text, estimates the probability of every possible continuation. chapter 0 →
- Learning rate Скорость обучения
- A coefficient controlling the scale of parameter updates. Too large can make training unstable; too small can make it slow. chapter 3 →
- Likelihood Правдоподобие
- The probability of observed data, viewed as a function of model parameters. Training can seek the parameters that maximise it. chapter 2 →
- Logit soft-capping Мягкий потолок логитов
- Logits pass through c·tanh(x/c) (c = 30 for Sprout-2): ordinary ones barely change, and none can rise past c. chapter 15 →
- Logits Логиты
- The model's raw scores for every token before softmax: any numbers, not probabilities. chapter 3 →
- LoRA LoRA
- Training low-rank corrections to selected matrices while freezing the original weights. Sprout’s adapters have 417,792 trainable parameters, about 2.4% of the model size. chapter 13 →
- Loss Функция потерь
- A numerical measure of error under a chosen rule. Training aims to reduce it, though an individual update can increase it. chapter 3 →
- Matrix rank Ранг матрицы
- How many independent directions a matrix holds. A rank-r matrix is a sum of r simple layers. chapter 13 →
- Muon Muon
- An optimizer that approximately orthogonalises momentum-based matrix updates. In Sprout’s short trials it achieved lower loss than AdamW for the same token budget. chapter 10 →
- Nat, bit Нат, бит
- Units of information: nats with the natural log, bits with log base 2. 1 nat ≈ 1.44 bits. chapter 2 →
- Neuron Нейрон
- In our teaching network, a weighted sum of inputs plus a bias, followed by an activation function such as tanh or ReLU. chapter 4 →
- Overfitting Переобучение
- The model fits training examples increasingly well while performance on new data stops improving or gets worse. chapter 5 →
- Parameter Параметр
- One number inside the model that training adjusts. Sprout has 17.3 million of them. chapter 0 →
- Perplexity Перплексия
- The exponential of average loss in nats. It expresses prediction loss on the scale of guessing among equally likely alternatives, rather than counting remaining tokens. chapter 2 →
- Proxy model Прокси-модель
- A small copy of the model used to compare settings cheaply before the big run: learning rates, the data mix, architectural details. chapter 15 →
- Quantisation Квантование
- Representing numbers with fewer possible values, usually using fewer bits. It saves memory; the effect on quality depends on the method and model. chapter 11 →
- Query, key, value Запрос, ключ, значение
- Queries and keys determine attention scores; values are mixed using the resulting weights. All three are computed from token representations. chapter 7 →
- Residual stream Остаточный поток
- The vector that flows through all blocks; every block adds its own correction to it. chapter 8 →
- RLHF, DPO RLHF, DPO
- RLHF uses human feedback in reinforcement learning. DPO trains directly on preference pairs without a separate reward-model training stage. chapter 17 →
- RMSNorm RMSNorm
- Divide vector coordinates by their root mean square, with a small stability term, then apply learned gains. chapter 8 →
- RoPE RoPE
- Rotary position embeddings: queries and keys are rotated by an angle that depends on the position. chapter 8 →
- Sampling Выборка (сэмплирование)
- Choosing a value at random, weighted by the probabilities, like rolling a loaded die. chapter 1 →
- Scaling laws Законы масштабирования
- Empirical formulas for how the loss falls as the model, the data and the compute grow. chapter 10 →
- SFT SFT
- Supervised fine-tuning: further training on prepared examples. Our chat setup scores assistant reply tokens and their end markers. chapter 12 →
- Softmax Softmax
- Turns logits into probabilities: exponentiate each one and divide by the sum. chapter 3 →
- Surprise (information) Удивление (информация)
- Minus the log of the probability: −log p. Unlikely events are more surprising. chapter 2 →
- SwiGLU SwiGLU
- A gated feed-forward layer: silu(xW₁) ⊙ xW₃. chapter 8 →
- System message Системное сообщение
- Text at the start of a conversation that sets rules or announces abilities, such as "you can run Python". A model follows it only as far as its training showed it such examples. chapter 16 →
- Temperature Температура
- The divisor of the logits before softmax: lower makes the model stick to its favourites, higher makes it more varied. chapter 11 →
- Thinking (reasoning before the answer) Мысли (рассуждение перед ответом)
- Steps the model writes before its answer, between <|think|> and <|/think|>. Every token is a little more computation, but it does not make the answer right by itself. chapter 16 →
- Token Токен
- A sequence unit represented by a vocabulary identifier: a word, word fragment, byte or special marker. chapter 6 →
- Tool call Вызов инструмента
- Instead of an answer the model writes a request to a program (for Sprout-2, Python code); the page runs it and hands the output back, and the model goes on. chapter 16 →
- Top-k, top-p Top-k, top-p
- Top-k keeps the k most likely tokens. Top-p keeps the shortest sorted prefix with total probability at least p. The retained probabilities are renormalised. chapter 11 →
- Transformer Трансформер
- A family of architectures built with attention, position-wise feed-forward layers and residual connections. Sprout uses a causal decoder-only transformer. chapter 8 →