More data

Sprout grew up on children's stories. 330 million tokens of fairy tales and everyday dialogue taught its 17 million numbers to tell a tale and chat about the weather. Now we plant a new seed. Sprout-2 has 125.1 million parameters, seven times as many, and a job is already waiting for it: later in the course it will become a tiny agent that reads code and runs commands.

A model is what it has read (chapter 9), so Sprout-2 needs a very different library: about 10.6 billion tokens of the modern English web, textbooks, code, questions and answers, commands and maths. But first, the most visible change: let's show Sprout-1 some code.

Sprout-1 meets code

Below, three tokenizers cut up the same text. GPT-2's 50,257 tokens were learned in 2019 from web pages; Sprout-1's 8,192, in chapter 6, from stories; Sprout-2's 32,768, from a sample mixed like this chapter's corpus. Start with the Python example, then try the others or type your own.

The Python example says it all.

• Sprout-1 has never seen code. "def" is two tokens, "de" and "f"; "values" is "val" and "ues". None of its 7,932 merges is whitespace or contains a newline, so every space of an indent is a token of its own: 67 tokens for a five-line function.
• GPT-2 knows the words, having read the web, but it too spends a token on every space of an indent: 16 of its 53 tokens are lone spaces.
• Sprout-2 needs 34. "):↵" ends a line in one token, an indent's spaces are one token, and so is "(values".

Plain English is a draw: 15 tokens for GPT-2 and Sprout-2, 19 for Sprout-1, whose stories had few oceans, so it spells "·o|ce|ans". And in one example Sprout-2 loses to everyone, on purpose: numbers. We'll come back to them.

As chapter 6 put it, a tokenizer reads in big, cheap pieces whatever its corpus had plenty of, and Sprout-2's corpus had plenty of code. So let's start with the corpus.

Why more, and what

In chapter 9 we chose children's stories on purpose: a 17-million-parameter model can't swallow the world, and trained on the web it would spread its few numbers too thin. Sprout-2 is seven times bigger and has a wider job, so its diet grows from one kind of text to six:

• Web and textbooks, 57.5% of the mix: explanations of everything, and ordinary English.
• Code, 17%, plus 2% of Markdown documentation: mostly Python and shell, the future agent's two languages.
• Questions and answers, 8%: code and commands explained in words, from Stack Overflow and other Stack Exchange sites.
• Commands and developer talk, 3.5%: examples for command-line tools, Ubuntu's IRC help channels, GitHub issues.
• Maths, 9%: step-by-step solutions, and word problems solved with short Python programs.
• Stories and dialogues, 3%: Sprout-1's own corpus, so the new plant can tell a story too.

How much text is enough? Chapter 9's rule of thumb, about 20 tokens per parameter, would give Sprout-2 some 2.5 billion. It trains on 9 billion, about 72 per parameter and 27 times what Sprout-1 read: the modern habit we met with Llama 3, where a small model that will be run many times is worth training far past the Chinchilla optimum. And to avoid rereading the same texts, the corpus needs at least that many tokens.

The mix

Here are all 31 sources, grouped by kind; a bar's length is a source's share of training tokens. Tap one to see what it is and its licence.

After cleaning, the corpus holds 10.57 billion tokens in 13.07 million documents; FineWeb-Edu alone brings 3.65 billion. Training doesn't pour the sources in one after another: every row of every batch comes from a source picked at random by its weight. The weights follow how many tokens we wanted from each source, so each is read roughly once, 0.7 to 1.1 times. The exception is tldr pages: only 1.16 million tokens, but superb short examples of command-line tools, so the mix shows them four times as often as their size says.

The cooldown mix

At the end of training the learning rate falls to zero: the cooldown from chapter 10. As the steps shrink, the model settles into what it reads last, a good moment to lean on what matters most. In train.py the cooldown is the last fifth of the steps, and the weights change: web pages drop to 0.6 of their usual weight, YouTube and IRC to 0.5; Python, shell, the Stack Exchange sites, FineMath and TinyGSM go up 1.5 times, tldr twice. On top come conversations in Sprout-2's chat format, 3% of the cooldown and none before: here the model first meets the special tokens it will need in chapter 16. Switch the widget to "Cooldown": code grows from 19% to 26%, Q&A from 8% to 12%, maths from 9% to 14%, and the web shrinks from 57% to 39%.

Licences

Sprout-2's weights, code and data list will be public, so everything it learns from must allow that. Every source in the mix is openly licensed:

Licence	Sources	Share of tokens	What it asks
ODC-BY 1.0	FineWeb-Edu, Cosmopedia, FineMath	48%	name the source
permissive, per file	all the code	18%	each file keeps its own licence; only MIT, Apache, BSD and the like were kept
CC-BY	DCLM, tldr, SODA, YouTube	16%	name the author
CC-BY-SA	Stack Exchange	9%	name the author; pass the texts on under the same licence
MIT	TinyGSM, SimpleStories	5%	keep the licence notice
open (Common Pile)	Ubuntu IRC, GitHub issues	4%	openly licensed or public domain
CDLA-Sharing 1.0	TinyStories	1%	pass the data on under the same terms

Almost all of them ask one thing: say where the data came from. So prepare.py writes a SOURCES.md next to the prepared data, with every source, its origin and its licence. One honest caveat: for FineWeb-Edu and DCLM the licence covers the collection, but the pages were gathered from the open web and their authors never chose a licence. That is the usual compromise of open models trained on the web. The Common Pile sources are stricter: every text in them is openly licensed or in the public domain.

A tokenizer for code

Why does GPT-2, which read 40 gigabytes of web text, spend a token on every space of an indent? Switch the widget at the top to "Chunks". Before merging, a tokenizer cuts text into chunks with a regular expression, and merges never cross a chunk's border (chapter 6). GPT-2's pattern glues a newline to the spaces after it, so every depth of indentation is a chunk of its own: "↵···", "↵·······". GPT-2 learned its merges from web pages, where such chunks are rare, so they fall apart into single characters.

Sprout-2's pattern was written with code in mind:

# Split text into chunks first, so a merge never crosses a chunk boundary: # contractions, words (with one leading space or punctuation mark), single # digits, runs of punctuation (with the newlines after them), newlines with the # spaces before them, spaces. Written without possessive quantifiers so that # JavaScript can use the same pattern. SPLIT = re.compile( r"""'(?:[sdmtSDMT]|ll|ve|re|LL|VE|RE)""" r"""|[^\r\n\p{L}\p{N}]?\p{L}+""" r"""|\p{N}""" r"""| ?[^\s\p{L}\p{N}]+[\r\n]*""" r"""|\s*[\r\n]+""" r"""|\s+(?!\S)""" r"""|\s+""")

• [^\r\n\p{L}\p{N}]?\p{L}+: a word may bring along one character that is not a letter, digit or newline: usually a space, as before, but also a bracket or a dot, so "(values" and ".append" can be single tokens.
• \p{N}: exactly one digit, without the space in front of it.
• ?[^\s\p{L}\p{N}]+[\r\n]*: punctuation takes the newlines after it, so "):↵", the end of a line of code, is one chunk.
• \s*[\r\n]+: a newline takes the spaces before it, never the ones after.
• \s+(?!\S)|\s+: runs of spaces, now standing alone: the indent. The last space goes with the next word, so "····if" becomes "···" and "·if".

The difference shows from Sprout-2's very first merge: two spaces. Four spaces came ninth, before "·the" (eleventh), and eight spaces 57th. 126 of its tokens are pure whitespace (runs of up to 64 spaces, runs of tabs, blank lines), and 498 merges contain a newline, like "):↵" and ":↵". Sprout-1's vocabulary has none of either. And since the pattern avoids possessive quantifiers, JavaScript runs it unchanged: the widget above reads it straight from Sprout-2's tokenizer.json.

Sixteen special tokens

Sprout-1 had four special tokens; Sprout-2 has sixteen, set aside in advance: <|endoftext|> between documents; <|user|>, <|assistant|>, <|end|> and a new <|system|> for conversations; <|think|> and <|/think|> around the model's own reasoning; <|tool_call|> and <|tool_result|> for calling a tool such as Python (all in chapter 16); three fim tokens for filling in the middle, later in this chapter; and four spares, <|reserved_0|> to <|reserved_3|>, for the agent to come.

In the "chat & thinking" example at the top, every marker is one token for Sprout-2, while Sprout-1 knows <|user|> but reads <|think|> as a handful of punctuation and letters. A spare token costs one row of the embedding table, 576 numbers, and saves us reshaping the table and the tokenizer later. In total: 256 bytes, 32,496 merges and 16 special tokens, exactly $2^{15} = 32{,}768$, so the ids still fit comfortably in uint16 (chapter 9's quiz).

How big a vocabulary?

train_tokenizer.py learned Sprout-2's merges from 603 million characters mixed like the corpus, in 75 seconds on the Mac, thanks to a heap that finds the most frequent pair without recounting everything. To choose the size, we trained three vocabularies and measured how many characters a token holds on unseen text; higher means fewer tokens.

Text	16k	32k	48k	GPT-2 (50k)
FineWeb-Edu	3.98	4.28	4.43	4.46
Python	3.33	3.62	3.77	2.17
shell	2.69	2.95	3.07	2.34
JavaScript	3.57	3.86	4.02	2.08
Stack Overflow	3.84	4.08	4.18	3.41
tldr pages	3.61	3.96	4.10	3.22
all of it together	2.95	3.13	3.22	2.86

GPT-2 still reads plain web English best, 4.46 characters per token against 4.28: it spent its 50 thousand tokens mostly on English words. On code the picture flips: Sprout-2 writes the same Python in 40% fewer tokens than GPT-2, and JavaScript in 46% fewer. As for size, 16k to 32k gains 6% on everything together, and 32k to 48k only another 3%, while the embedding table would grow by $16{,}384 \times 576 \approx 9.4$ million numbers. At 32k it already holds 18.9 million, 15% of the model. So 32k it is.

The price: numbers

Now the example Sprout-2 loses. "In 2024 the price rose from 1,250 to 13,999 dollars": GPT-2 needs 15 tokens, Sprout-1 23, Sprout-2 26. Sprout-2 writes every digit as a token of its own and even keeps the space before a number apart, so "·2024" is five tokens: "·", "2", "0", "2", "4". None of its 32,496 merges contains a digit.

We pay that price on purpose. For GPT-2, "250", "999" and "·13" are single tokens: to add numbers, a model must learn facts about thousands of arbitrary number-tokens, cut differently in different contexts. With one digit per token, every number is written the same way, digit by digit, and column addition becomes a pattern to learn once. Sprout-2 will need it for the word problems of chapter 16.

Sprout-2 spends five tokens on "·2024", where GPT-2 spends one. Why is that a deliberate choice rather than a flaw?

Sprout-2's vocabulary is too small to fit numbers With one digit per token every number is written the same way, digit by digit, which makes arithmetic easier to learn Digits are rare in Sprout-2's corpus, so no merges were learned for them

The pattern makes every digit a chunk of its own (\p{N}), so BPE can never merge digits, however often they appear. It costs a few tokens per number, but units line up with units and tens with tens, and the model doesn't have to memorise thousands of number-tokens.

Cleaning at scale

In chapter 9 the cleaner threw almost nothing away: synthetic stories are tidy. Now there are 31 sources and 13.19 million documents, from web pages to shell scripts. prepare.py sends every raw file, many at once, through the same steps: clean, filter, check against benchmarks, scrub secrets, cut for fill-in-the-middle, encode. Then it drops exact duplicates across all sources and sets aside a slice of each for validation.

The first step already shows why code needs care. Chapter 9's cleaner squeezed every run of spaces down to one: harmless for a story, fatal for Python, where the indent is the program. So prepare.py knows three kinds of text. Prose gets plain quotes, plain dashes and squeezed spaces; code keeps every space; mixed sources such as Q&A, Markdown and IRC get plain quotes but keep their spaces.

Of 13,191,986 documents read, 13,074,403 remain: only 0.89% went. The big web collections arrive already filtered by their makers, and our filters are deliberately simple. Two thirds of what went are exact duplicates. Then 23,706 documents are mostly in another script (under 90% Latin letters, 85% for code), most often Python files commented in another language. 8,347 are machine-made code, useless for learning to write it: minified scripts, generated tables, files that say "do not edit" at the top. 3,472 contained benchmark questions (see below), and about two thousand each were too short or too long.

Duplicates

Finally, prepare.py fingerprints each document (MD5 of its lower-cased text with whitespace squeezed), walks through all 31 sources in a fixed order and keeps only the first document with each fingerprint. 77,713 copies went: 52 thousand from FineWeb-Edu, 19 thousand from TinyGSM, 5.5 thousand from Cosmopedia. It is chapter 9's exact deduplication, and it still misses near-duplicates; the MinHash note there explains how big corpora catch those.

Keys and addresses

Code on the internet is full of things that should never have been published: API keys, access tokens, passwords, and e-mail addresses everywhere. A model can memorise such strings and repeat them to anyone. prepare.py keeps these documents (a configuration file is still a good example of one) but replaces the dangerous strings with "placeholders the model may learn to write instead of real secrets", as the comment in the code says.

E-mail addresses become name@example.com, except those that are examples already, GitHub's no-reply addresses and git@… (in git@github.com it is part of a command, not a person): 219,810 addresses were replaced. Keys are recognised by their real formats: AWS keys start with AKIA and become AKIAIOSFODNN7EXAMPLE, the example from AWS's own documentation; GitHub tokens start with ghp_ and its relatives, Slack tokens with xox, many API keys with sk-; a private key keeps only its BEGIN and END lines. A long random-looking value right after a name like api_key goes too. Passwords stay: in code they are nearly always examples. In all, 4,172 secrets were replaced, most in Python (1,700), on Stack Overflow (568) and in JavaScript (562).

The first version of the scrubber was greedier and caught too much. URLs like …/task-to-replace-… matched the rule for API keys: look inside "task-to-replace" and you'll find "sk-" followed by a long run of letters and dashes. Shell variables like ${PASSWORD} were taken for passwords. Try both examples in the widget. Now the key rule wants no part of another word right before "sk-", plus a digit, a capital letter and at least 32 characters; a value after a key-like name must look random: 16 characters or more, with a digit. Every filter has false positives, and you find them only by reading what it caught, not just counting it.

Here is the code from prepare.py; the next section puts it to work.

from collections import Counter import regex as re # placeholders the model may learn to write instead of real secrets EMAIL = re.compile(r'\b([A-Za-z0-9._%+-]+)@([A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)*\.[A-Za-z]{2,})\b') EMAIL_KEEP = {'example.com', 'example.org', 'example.net', 'users.noreply.github.com'} SECRETS = [ (re.compile(r'AKIA[0-9A-Z]{16}'), 'AKIAIOSFODNN7EXAMPLE'), (re.compile(r'gh[pousr]_[A-Za-z0-9]{36,}'), 'YOUR_GITHUB_TOKEN'), (re.compile(r'xox[baprs]-[A-Za-z0-9-]{10,}'), 'YOUR_SLACK_TOKEN'), (re.compile(r'(?<![A-Za-z0-9_-])sk-(?:proj-)?(?=[\w-]*\d)(?=[\w-]*[A-Z])[A-Za-z0-9_-]{32,}'), 'YOUR_API_KEY'), (re.compile(r'(-----BEGIN ([A-Z ]*)PRIVATE KEY-----).*?(-----END \2PRIVATE KEY-----)', re.S), r'\1\nYOUR_PRIVATE_KEY\n\3'), ] # a long random-looking value right after a key-like name (passwords in code are nearly always examples: kept) ASSIGNED = re.compile(r'''((?:api[_-]?key|apikey|secret[_-]?key|client[_-]?secret|access[_-]?token|auth[_-]?token)''' r'''["']?\s*[:=]\s*["'])((?=[A-Za-z0-9+/=_-]*\d)[A-Za-z0-9+/=_-]{16,})(["'])''', re.I) def scrub(text, counts): def email(m): if m.group(2).lower() in EMAIL_KEEP or m.group(1).lower() == 'git': return m.group(0) counts['emails'] += 1 return 'name@example.com' text = EMAIL.sub(email, text) for pattern, placeholder in SECRETS: text, n = pattern.subn(placeholder, text) counts['secrets'] += n def assigned(m): counts['secrets'] += 1 return m.group(1) + 'YOUR_API_KEY' + m.group(3) return ASSIGNED.sub(assigned, text)

No peeking at the exam

Sprout-2 will be graded: on common sense (HellaSwag, PIQA, WinoGrande), school science (ARC, OpenBookQA, SciQ), reading comprehension (BoolQ), and writing Python that works (MBPP, HumanEval, GSM8K's word problems). A test measures skill only if the model has never seen its questions. But questions leak onto the web, into quiz pages and study guides, and a model that read them earns a score for memory.

So before a document is kept, prepare.py compares it with every benchmark. It cuts every benchmark text into all its runs of 13 consecutive words, lower-cased and ignoring punctuation, and keeps their hashes in a set. Then it does the same with each document; a single shared run, and the document goes. Looking a hash up in a set takes the same time however large the set (chapter 9), so the check stays cheap even for 13 million documents.

3,472 documents went: 1,386 from FineWeb-Edu, 646 from FineMath, 574 Python files, 207 from DCLM, 183 from Cosmopedia. In the examples, a maths quiz page carries a GSM8K test problem whose first 28 words match; a cosmetic clinic's page in FineWeb-Edu repeats, word for word, the definition of a keloid scar from the WikiHow article behind a HellaSwag question; and a sentence about the Coriolis effect sits both on a DCLM page and in an ARC question.

The fourth is a false alarm. A Python file that finds the closest pair of points contains the same double loop as an MBPP solution that counts inversions: for i in range(n): for j in range(i + 1, n): if. Punctuation doesn't count, so this ordinary loop is exactly 13 words: for, i, in, range, n, for, j, in, range, i, 1, n, if. In prose, 13 words in a row almost never match by chance; in code, they are one line of a common idiom. So the check is crude for code, and some of the 574 dropped Python files were surely innocent. But that is under 0.1% of the Python, a price we accept for a clean exam.

Let's run both filters on a small file: a closest-pair function with a forgotten key in it. (The previous cell runs first by itself.)

# prepare.py's ngrams(), unchanged: every run of 13 words, as a hash WORD = re.compile(r'\w+') def ngrams(text, n=13): words = WORD.findall(text.lower()) return (hash(tuple(words[i:i + n])) for i in range(len(words) - n + 1)) mbpp = ("def get_Inv_Count(arr,n): \n inv_count = 0\n for i in range(n): \n" " for j in range(i + 1,n): \n if (arr[i] > arr[j]): \n" " inv_count += 1\n return inv_count") bad = set(ngrams(mbpp)) # prepare.py fills this set from every benchmark text fake_key = 'sk-proj-' + 'NotARealKey0' * 3 doc = f'''import math API_KEY = "{fake_key}" OWNER = "jane.doe@mycompany.io" def closest(points, n): best = float('inf') for i in range(n): for j in range(i + 1, n): if math.dist(points[i], points[j]) < best: best = math.dist(points[i], points[j]) return best ''' counts = Counter() print(scrub(doc, counts)[:80]) print(dict(counts)) shared = [h for h in ngrams(doc) if h in bad] print(len(shared), 'run(s) of 13 words shared with MBPP:', 'the document goes' if shared else 'it stays')

In prepare.py the benchmark check comes before the secret scrubber, so this file would never even reach it.

Why does the 13-word check throw out innocent code more often than innocent prose?

Code files are longer than web pages Punctuation doesn't count, so one common line of code, a loop or a call, is already 13 "words" that many programs share Benchmarks contain more code than prose

In prose, 13 identical words in a row are almost always a real copy. Code is built from a few keywords and short names, and the brackets between them are ignored, so ordinary idioms reach 13 words quickly. A fairer check for code would need longer runs or whole functions.

Fill in the middle

An agent rarely writes a file top to bottom: it opens an existing one and changes a few lines in the middle. A model trained on text read left to right can only continue: when it writes, it knows the beginning of the file but never its end. Yet the end matters: the missing line must give the code below what it expects.

The fix is surprisingly simple. Cut a file at two random points into a prefix, a middle and a suffix. Reorder the pieces and mark them with three special tokens: prefix first, then suffix, then middle. A model reading left to right now reaches the middle having seen both the beginning and the end. It learns to fill the gap, and nothing in the model has to change.

In prepare.py it takes six lines:

if source in CODE and len(text) > 200 and random.Random(h).random() < FIM_RATE: rng = random.Random(h + 1) a, b = sorted(rng.randrange(len(text)) for _ in range(2)) pre, mid, suf = (fast.encode_ordinary(t) for t in (text[:a], text[a:b], text[b:])) doc = [eot, _w['fim'][0], *pre, _w['fim'][1], *suf, _w['fim'][2], *mid] counts['fim'] += 1

Only code files over 200 characters take part, and only 30% of them (FIM_RATE); the choice and the cuts depend on the document's hash h, so a rerun makes the same cuts. 511,834 files were cut, 29% of the code files kept (Markdown doesn't take part). The rest stay in order, so the model still learns to write code from the top down. A cut can fall inside a word, and since each piece is tokenized on its own, the tokens at the seam change: try cutting "json" in half.

How much code?

One question is left: why 17% code? More code makes a better programmer, but every token of code is a token not spent on English, and the agent has to understand the person as well as the program. Rather than guess, we trained small copies of Sprout-2, proxies: 14.3 million parameters and 100 million tokens each, about an hour apiece on the Mac. Three differed only in their share of code: half the plan, the plan, and one and a half times the plan. Each was then measured on held-out text from every group of sources.

The result is lopsided. From half the plan to 1.5 times it, the loss on code falls by 0.24 nats and the loss on web text rises by only 0.06: much better code, slightly worse English. But the first step buys more than the second. From half to the plan, code improves by 0.16; beyond it, by only 0.08, while the price in web English grows from 0.02 to 0.04. Stories and maths pay a little too, 0.02 to 0.04 nats at each step. Returns shrink on one side and costs grow on the other; the planned 17% is a compromise between them, and the cooldown leans on code anyway.

Proxies are almost nine times smaller than Sprout-2 and read 90 times less, so their numbers show a direction, not a forecast. The learning rates and the document mask of chapter 15 were chosen on proxies too.

Sprout right now

Sprout-2 isn't in your browser yet: it is still growing. But we can read its diary: every now and then during training it continues the same five prompts. After 200 steps of 524,288 tokens it had read 105 million, a third of everything Sprout-1 ever read, and it already writes English with the rhythm of a story and Python with its indents in the right places, though its code doesn't compute anything yet. By step 1,000 the story has a heroine and a plot, and the function has a docstring. By step 6,000 it has read a third of its 9 billion tokens. These are its first hours; the next chapter shows the model that reads all this, and how it grows to the end.