How much memory it takes

The catalogue of M5+ earthquakes over eleven years is 19,073 lines, a megabyte and a half of text. Each line is a time, latitude, longitude, depth, magnitude, magnitude type and place, about eighty bytes. Read it in Python in the most ordinary way, list(csv.DictReader(f)), and it takes more than twelve megabytes in memory, eight times the file. Below is the same catalogue stored seven ways in Python and in Go.

The heaviest way is dictionaries: more than six hundred bytes per earthquake, about two hundred and seventy of them the hash table of a seven-key dictionary itself, and the rest seven separate strings. Tuples and objects with __slots__ are half as heavy, columns a quarter. In Go everything is several times more compact, but even there a slice of map[string]string dictionaries weighs almost three times as much as a slice of structs. Move the slider: at a million records Python’s dictionaries are already more than half a gigabyte, and at a hundred million a laptop will not be enough.

The idea

The weight of data is made of two parts: the values themselves and what they are wrapped in. In Go a value is almost always its bytes: a number is 8 bytes, a struct the sum of its fields with small gaps. In Python every value is a separate object with a header: a number weighs 24–28 bytes instead of 8, a string 41 bytes plus the text. The estimate comes down to a multiplication: bytes per record × the number of records. The bytes per record can be counted from the table below or measured on a thousand records.

Note the word “records”. What matters is not how much the file weighs but how many objects it will break into in memory: a thousand long strings and a million short numbers weigh quite differently, even if the files are the same size.

What things weigh in Python

On 64-bit CPython 3.13 the function sys.getsizeof shows:

Python keeps the numbers from −5 to 256 in a single instance, and lists of them are cheaper: they hold only pointers. Identical strings are not shared: csv creates a new string for every value. In our catalogue the magnitude type has only nine different values, yet there are nineteen thousand strings “mww” and “mb” in memory. sys.intern or a replacement dictionary leaves one copy of each and saves more than eight hundred kilobytes, a quarter of the columnar version.

The figures in the table can be checked with two instruments. sys.getsizeof tells about the object itself but not about the ones it refers to, so the weight of a data structure is a recursive sum over all objects, with shared ones counted once. tracemalloc sees every memory allocation. On all four ways of storing the catalogue they agree to within hundredths of a megabyte.

What things weigh in Go

In Go an int or float64 is 8 bytes without any headers. A string is a 16-byte header (a pointer to the text and a length) plus the text itself; a slice is 24 bytes (pointer, length, capacity) plus the array. A struct is its fields in a row, but not packed tight: each field lies at an address that is a multiple of its alignment, and there may be empty bytes between fields.

Six fields, three bool, a float64, an int32 and an int64, take 40 bytes in a “however it happened” order, 17 of them empty. If the fields are laid from largest to smallest, it comes to 24. Over a million such records that is a 16-megabyte difference from the order of lines in one definition. How alignment works and why the order “largest to smallest” is optimal is worked through in the problem below.

A map[string]string dictionary in Go is cheaper than Python’s, but still heavy: each of the seven pairs has two string headers of 16 bytes, plus the service part of the table. In the measurement more than half the weight of this way is string headers and the table, not the text.

Code

The catalogue four ways in Python, the weight by recursive sum and by tracemalloc, and the weight of individual objects:

In Go the weight is measured on the heap: collect garbage, note the heap size, build the data, collect garbage again and subtract. Strings are copied explicitly, otherwise all fields would point into a shared piece of memory read from the CSV, and we would be measuring the wrong thing. The last lines are the size of a struct with two orders of fields.

How to estimate

1. Count the records and fields. A million records of seven fields each is seven million values.
2. Estimate the bytes per value. Python: a number is 24–28 bytes, a string 41 plus its length, a record container from 40 to 300. Go: a number is 8, a string 16 plus its length.
3. Multiply and add headroom. Lists and slices grow with spare room of up to a quarter and up to half the capacity; while being built, data sometimes exists in two copies.
4. Measure on a thousand records and multiply: this is more accurate than any table and takes a minute.

If the estimate does not fit in memory, there are three ways out. Change the representation: columns instead of objects, numbers instead of strings (time as eight bytes of seconds instead of a 60-byte string), one instance instead of a thousand identical ones. Change the language or library: numpy and pandas store columns the way Go does. Or do not hold everything at once: read the stream in parts, as external sorting does. How the layout in memory also affects speed is told in the article “Cache-friendly data”.

Problems

The problem is to compute the size of a Go struct the way the compiler does, and to find the order of fields at which it is lightest. The answers have been checked against unsafe.Sizeof in real Go.