Character Encoding
The mapping between characters and bytes, and the source of a specific family of invisible bugs.
Character encoding is the correspondence between characters and their byte representation. Unicode assigns a code point to every character; UTF-8 encodes those code points as one to four bytes, is backward-compatible with ASCII, and has won.
Even with the encoding war settled, several distinctions still cause bugs. A code point is not a character a person would recognize: emoji with modifiers, and accented letters written as a base plus a combining mark, span several. So "length" has at least three meanings — bytes, code points, and grapheme clusters — and string operations that split or reverse text will produce nonsense if they use the wrong one. Normalization matters too, since visually identical strings can differ in bytes.
The sharpest practical hazard is control characters, and particularly the null byte. A file containing a real null is still valid text to a compiler and a formatter, renders as innocuous whitespace in most viewers, and causes search tools to treat the file as binary and suppress all matches silently. A search that returns nothing therefore looks like an absent symbol rather than a corrupted file. Inspecting the bytes is the only way to see it. See Silent Failure.
Terminal alignment is a related concern: East Asian characters occupy two columns and combining marks occupy none, so column arithmetic must use display width. See Terminal User Interface.
See also4
Token
The sub-word unit a language model actually reads and writes; the unit of cost and of context.
Agents & Language Models17 connections
Shell
The interactive command interpreter, and a programming language with unusually sharp edges.
Systems & Tooling15 connections
Terminal User Interface
An interface rendered as text in a terminal, and the design conventions that make command-line tools humane.
Graphics & Rendering12 connections
Silent Failure
A failure that produces no signal, so the absence of an error reads as success.
Method63 connections
Related6
Nearby in the graph rather than deliberately chosen. Looser, sometimes surprising.
Linked from7
- Automatic Speech RecognitionAgents & Language Models
Turning audio into text, and the reasons real-world accuracy is set by the signal more than by the model.
- ShellSystems & Tooling
The interactive command interpreter, and a programming language with unusually sharp edges.
- Terminal User InterfaceGraphics & Rendering
An interface rendered as text in a terminal, and the design conventions that make command-line tools humane.
- Time ZoneSystems & Tooling
Civil time is a political layer over solar time, and its rules change often enough to be data.
- TokenAgents & Language Models
The sub-word unit a language model actually reads and writes; the unit of cost and of context.
- Video Game LocalizationPlay & Games
Translating a game plus adapting it for a new market, and the drift that adaptation introduces.
- XDG Base Directory SpecificationSystems & Tooling
A convention placing configuration, data, cache, and state in defined directories rather than the home directory.