Encoding, Serialization & Compression

From ASCII to UTF-8

Every character you see on screen is stored as a number. The mapping from numbers to characters is called a character encoding. Getting it wrong produces mojibake — garbled text like é instead of e.

ASCII

The original 7-bit encoding. Maps 128 characters (0-127): English letters, digits, punctuation, and control characters. Every byte's top bit is 0. Simple, but only covers English.

Unicode

A universal character set assigning a unique code point (e.g., U+0041 = A, U+1F600 = a smiley) to every character in every script. Unicode defines what characters exist; encodings define how to store them as bytes.

UTF-8

The dominant encoding on the web. Uses 1 to 4 bytes per character with a clever variable-length scheme:

UTF-16 and UTF-32

UTF-16 uses 2 or 4 bytes per character. It's used internally by Java, JavaScript, and Windows. Characters outside the Basic Multilingual Plane need surrogate pairs (two 16-bit units).

UTF-32 uses exactly 4 bytes per character. Simple indexing but wasteful — rarely used for storage or transmission.

Byte Order Mark (BOM)

A special code point (U+FEFF) placed at the start of a file to indicate encoding and byte order. Essential for UTF-16 (which needs to signal big vs. little endian), but optional and generally discouraged for UTF-8.

Integers, Floats & Endianness

Integer Representation: Two's Complement

Modern CPUs represent signed integers using two's complement. The most significant bit is the sign bit (0 = positive, 1 = negative). To negate a number: flip all bits, then add 1.

 5 in 8-bit:  00000101
-5 in 8-bit:  11111011  (flip → 11111010, +1 → 11111011)

This gives a range of -128 to +127 for 8-bit, and -2,147,483,648 to +2,147,483,647 for 32-bit. The beauty: addition and subtraction work identically for signed and unsigned values — no special hardware needed.

IEEE 754 Floating Point

Floats store real numbers as sign × mantissa × 2^exponent. A 64-bit double has 1 sign bit, 11 exponent bits, and 52 mantissa bits.

Endianness

Multi-byte values can be stored with the most significant byte first (big-endian, aka network byte order) or least significant byte first (little-endian, used by x86/ARM). The integer 0x01020304:

Big-endian:    01 02 03 04
Little-endian: 04 03 02 01

Network protocols (TCP/IP) use big-endian. Most modern CPUs use little-endian. Serialization formats must specify or detect byte order to avoid silent data corruption.

Base64, Base32 & Hex

Base encodings represent arbitrary binary data using a restricted alphabet of printable characters. They're essential when binary data must travel through text-only channels (email, JSON, URLs).

Base64

Encodes 3 bytes (24 bits) into 4 characters from a 64-character alphabet (A-Z, a-z, 0-9, +, /). Adds ~33% overhead. Padding with = aligns the output to a multiple of 4 characters.

"Man" → TWFu

M = 77 = 01001101
a = 97 = 01100001
n = 110 = 01101110

010011 010110 000101 101110
  T      W      F      u

Base32 & Hex

Base32 uses 32 characters (A-Z, 2-7). Case-insensitive and avoids ambiguous characters. Used in TOTP codes and some file systems. 60% overhead.

Hex (Base16) maps each byte to two characters (0-9, a-f). 100% overhead but trivial to read and debug. Used for hashes, colors, and memory addresses.

Turning Objects into Bytes

Serialization converts in-memory data structures into a format that can be stored or transmitted. The choice of format affects performance, interoperability, and developer experience.

Protocol Buffers Deep Dive

Protobuf assigns each field a numeric tag. The wire format encodes tag + type + value, skipping unset fields entirely. This makes it compact and allows schema evolution.

// person.proto
syntax = "proto3";

message Person {
  string name = 1;    // field tag 1
  int32  age  = 2;    // field tag 2
  string email = 3;   // field tag 3
}

Changing Formats Without Breaking Things

In distributed systems, producers and consumers of data are updated independently. Schema evolution rules ensure old readers can handle new data and vice versa.

Compatibility Types

Evolution in Practice

// v1: original schema
message Event {
  string id = 1;
  string type = 2;
}

// v2: backward + forward compatible
message Event {
  string id = 1;
  string type = 2;
  int64 timestamp = 3;   // NEW — old readers ignore it
  // field 4 reserved     // REMOVED field — tag retired
}

Making Data Smaller

Compression trades CPU time for smaller data. The right algorithm depends on whether you need speed (real-time streaming) or maximum size reduction (archival storage).

HTTP Compression

Browsers and servers negotiate compression via HTTP headers. The client advertises what it supports; the server picks the best match.

# Request
GET /api/data HTTP/1.1
Accept-Encoding: gzip, deflate, br

# Response
HTTP/1.1 200 OK
Content-Encoding: br
Vary: Accept-Encoding
Content-Type: application/json

[compressed body]