Lesson №06 20th century 18 min
Binary, hex and Base64
A computer knows no letters, only zeros and ones. Let’s see how text turns into bytes, what hex and Base64 are for, and why a string that looks like gibberish isn’t a cipher yet.
In this lesson
- what bits and bytes are, and how counting in binary works
- why programmers like hexadecimal
- how letters become numbers: ASCII, code pages, UTF-8
- how Base64 works, and why it hides nothing
An auditor opens the config file of a small online shop. Next to “admin password” it says cXdlcnR5MTIz. “Our passwords are encrypted,” the developer says proudly. The auditor copies the string, pastes it into the first online decoder that comes up, and a second later the screen shows qwerty123. There was no cipher. There was Base64, a way to write down data, not to hide it.
This story happens all the time, because encoded data looks like gibberish. But gibberish isn’t the same as a secret. In this lesson we follow text as it turns into numbers, and numbers into different notations: binary, hexadecimal, Base64. And we learn to tell an encoding from a cipher at a glance.
Bits and bytes
In the last lesson the telegraph sent two states: current on, current off. A computer works the same way. The smallest piece of information is a bit: zero or one, off or on. A single bit can answer a yes-or-no question, and nothing more.
To tell more things apart, bits are grouped. Eight bits make a byte. Eight switches have 2 × 2 × … × 2 = 256 positions, so a byte holds a number from 0 to 255. Each switch has its own weight: the rightmost is worth 1, the next 2, then 4, 8, 16, 32, 64, and the leftmost 128. The number in the byte is the sum of the weights of the switches that are on.
That’s the binary system: the same way of writing numbers as decimal, only with two digits instead of ten, and each place worth twice its right-hand neighbour instead of ten times. The number 65 in binary is 01000001: 64 + 1.
You count in binary the way an odometer counts, only it rolls over after 1 instead of after 9: 0, 1, 10, 11, 100, 101, 110, 111, 1000. Try it in the widget with the “+1” button and watch the ones carry to the left.
Hexadecimal
Binary is honest but long: eight characters per byte, and easy to lose your place in. Decimal is shorter but awkward: the number 200 doesn’t show you which bits are on. Programmers settled on something in between, the hexadecimal system, or hex. It has 16 digits: 0–9 and the letters A–F, where A means 10 and F means 15.
The beauty of hex is that one hex digit is exactly four bits. A byte splits in half, and each half becomes one digit. 65 = 0100 0001 = 41. 255 = 1111 1111 = FF. So any byte in hex is exactly two digits, from 00 to FF, and you can read the bits straight off them. Look at the widget above: the two hex digits are underlined in different colours, one for each half of the byte.
Hex is everywhere once you notice it. Web colours are written like #1D4E89: three bytes for red, green and blue. Network card addresses look like 00:1A:2B:…. If you see pairs of characters from 0–9 and a–f, you’re almost certainly looking at bytes in hex.
How letters became numbers
A byte is a number. To store text, people have to agree which number stands for which letter. Such an agreement is called a character encoding. The best known is ASCII, an American table from 1963. It has 128 characters: Latin letters, digits, punctuation and control codes such as “new line”. A is 65, B is 66, lowercase a is 97, a space is 32, the digit 0 is 48.
Notice the neat trick in that table: an uppercase letter and its lowercase twin differ by exactly 32, which is a single bit. A is 01000001, a is 01100001. Flip the 32 switch in the widget and the letter changes case.
ASCII only needed seven bits, so the top half of the byte, numbers 128 to 255, was free. Everyone filled it differently. Western Europe put é, ü and ß there; Russia put Cyrillic; Greece put Greek. IBM PCs had their own table full of line-drawing characters, Windows had Windows-1252. The same byte meant different things in different tables. Open a file with the wrong one and the text turns into garbage. That garbage even has a name, mojibake, from the Japanese for “character transformation”: Japanese users suffered from it more than anyone.
You have probably met mojibake yourself. The word “café” saved in UTF-8 and opened as Windows-1252 becomes “café”. A curly apostrophe in “it’s” becomes “it’s”. Each of those odd pairs and triples is one character whose bytes were read with the wrong table.
Unicode and UTF-8
To end the mess, the world built Unicode: one table for everything. Every character gets its own number. Latin A is 65, same as in ASCII; the Russian letter я is 1103; there are tens of thousands of Chinese characters. A number is written as U+ followed by hex: я is U+044F.
But numbers that run past a million don’t fit in one byte. How do you store them as bytes? The most popular answer is UTF-8. Ken Thompson and Rob Pike designed it in 1992, sketching it out one evening on a placemat in a New Jersey diner. The rule: ASCII characters take one byte, exactly as before; Greek and Cyrillic take two; most others take three or four. The first bits of every byte are markers: they say how many bytes the character has, and which bytes continue the one before.
Look at ё: its number, 1105, doesn’t fit in seven bits, so UTF-8 uses two bytes, D1 91. The letter H, on the other hand, is still the single byte 48, exactly as in ASCII. This is why UTF-8 won: every old English text is already valid UTF-8, and a program that only knows ASCII can pass UTF-8 along without breaking it.
A note arrives with nothing but eleven groups of zeros and ones. Where is the meeting?
01101110 01101111 01110010 01110100 01101000 00100000 01110100 01101111 01110111 01100101 01110010
Each group is one byte. Set its bits in the switch widget above and read the letter it shows.
Almost every byte starts with 011: that’s the range from 96 to 127, where the lowercase letters live. The odd one out, 00100000, is 32: a space.
The first byte is 64 + 32 + 8 + 4 + 2 = 110, the letter n. The second is 111, o. Going on: r, t, h, space, t, o, w, e, r. The note says north tower.
The Base64 converter reads binary too: paste the line and it shows the text straight away.
Base64: bytes as letters
Now we have bytes. They can sit in a file or travel over a network. But some places only accept text, and the plainest kind at that: Latin letters, digits and a couple of symbols. Email was built like that. Its protocols were designed for English messages, and a byte with the top bit set could get mangled on the way. So how do you send a photo or a zip file?
The answer is Base64. It takes any bytes and writes them with 64 “safe” characters: 26 uppercase Latin letters, 26 lowercase, 10 digits, plus “+” and “/”. The result is text that survives any mail server, a browser’s address bar or a settings file. Email attachments still travel this way.
Why 64? Because 64 = 2 × 2 × 2 × 2 × 2 × 2: one Base64 character carries exactly six bits. Bytes have eight. The smallest number divisible by both 6 and 8 is 24. So Base64 works in threes: it takes three bytes (24 bits), cuts them into four pieces of six bits, and replaces each piece with a character from its table. The piece 000000 is A, 000001 is B, and so on up to 111111, which is “/”.
The classic example is the word “Man”: the bytes 4D 61 6E become exactly four characters, TWFu. Now look at the end of “Man!”. Four bytes are one full group and one leftover byte. The last group is filled out with zeros, and every six that holds no real bits at all is written as =. That tells the receiver how many bytes there really were: one “=” means two leftover bytes, two “==” mean one. It’s also why Base64 is always about a third longer than the original: every three bytes become four characters.
There is a variant of Base64 for web addresses, where “+” and “/” are swapped for “-” and “_” so the string doesn’t break a link. The login tokens many websites hand out are written in it. And Base32 cuts the bits into fives and gets by with 32 characters, the capital letters and the digits 2–7, so you can read it out over the phone without worrying about upper and lower case.
A program turns 10 bytes into Base64, with the usual “=” padding. How many characters long is the result? Answer with a number.
Base64 eats bytes three at a time, and every group, even an unfinished one, becomes four characters.
Ten bytes are three full groups and one byte left over. Type any ten letters into the Base64 machine above and count the blocks.
Three full groups take 9 bytes and give 12 characters. The tenth byte opens a fourth group, which also becomes four characters: two real ones and “==”. In total 16.
The general rule: round the number of bytes up to a multiple of three, divide by three, multiply by four.
An encoding is not a cipher
Back to the developer from the start of the lesson. His mistake was mixing up two different things. A cipher depends on a key: without the key you can’t read the text, even if you know the method. An encoding has no key. Its rules are published, and anyone who recognises the format can decode the string in a second. Binary, hex, Base64, Morse code: these are all ways of writing things down, not ways of protecting them.
The good news is that encodings are easy to recognise. A string of zeros and ones in groups of eight is binary. Pairs of characters from 0–9 and a–f are hex. Latin letters in both cases mixed with digits, a length that divides by four, maybe one or two “=” at the end: almost certainly Base64. The cipher identifier runs these checks for you, and the Base64 converter peels the layers off one by one when data has been wrapped several times, say text in Base64, that in hex, and the hex in binary.
An encoding changes how data is written, not what it says. It has no key, and anyone can reverse it. Binary, hex and Base64 are different ways of writing the same bytes. Only a cipher can hide them.
A database stores a user’s password as cXdlcnR5MTIz. Is it protected?
Base64 is an encoding, not a cipher. This string turns into qwerty123 in a fraction of a second. Passwords are stored differently: they go through a special hash function, which we’ll meet in the lesson on hashes.
Why is one byte always written as exactly two hex digits?
Sixteen is 2⁴, so each hex digit describes exactly four bits. An eight-bit byte is two halves, two digits: from 00 to FF.
What to take with you
- A bit is a zero or a one; a byte is eight bits, a number from 0 to 255. Binary shows the bits; hex shows the same bits four to a digit.
- A character encoding is an agreement about which number means which letter. ASCII only knows English; UTF-8 writes all of Unicode, English in one byte per letter, most other alphabets in two or more.
- Base64 writes any bytes with 64 safe characters: three bytes become four characters, and “=” at the end marks the shortfall.
- An encoding isn’t a cipher: it has no key. If a string looks like Base64 or hex, anyone can read it.
A cipher makes a message unreadable. But there is a sneakier trick: making sure nobody sees there is a message at all. Letters hidden in the lowest bits of pixels, a picture inside a sound, an innocent sentence with a secret in it. Next: steganography →
Tools for this lesson
Everything from the lesson, working: paste your own text.
- Base64, hex and binary Convert text to and from Base64, Base32, hex, binary and decimal bytes (UTF-8).
- Cipher identifier Paste a ciphertext and the analyser suggests what it may be: Caesar, Vigenère, Base64, Morse, hex…
- XOR cipher online XOR text or bytes with a key in hex, Base64 or text, and brute-force a single-byte key.