go / syntax highlighting
highlight colors source code for server-rendered HTML. It is a hand-written, line-oriented scanner per language over a shared token type.
highlight.Code(w, "main.go", src) // or "go", from a markdown fence
highlight.Diff("main.go", patch) // a unified-diff patchIt colors this blog's code fences, built by cmd / blog, and the diff view in cmd / cibot.
Curation is the product
Scanners exist for Go, SQL, CSS, JavaScript, C, Ruby, JSON, Lua, HTML, Markdown, shell, and hml. A format with no scanner is escaped and rendered uncolored, so callers can mix without checking first.
It is fast and small because it is deliberately incomplete. A word a keyword list misses comes out uncolored rather than wrong.
This blog ran Chroma before. Chroma carries hundreds of lexers, a style system, and a transitive regexp2, and it still had no lexer for hml, so those blocks rendered as plain text. The scanners were already written in cibot for its diff pages.
Classes are the contract
A short list of classes comes out of the code path: k keywords, n
names, s strings, m numbers, c comments. A diff row carries one
more: gi inserted, gd deleted, gu hunk header, gc context.
There is no shipped palette. Those names are the whole contract with a stylesheet, and colors belong to the consuming site. This blog styles them twice, once for the light palette and once for Catppuccin Frappe, the flavor my editor uses, so a snippet here looks like the file it came from.
Two layers
Tokens are the lower layer. A scanFunc reads one line, given the state
the line before it ended in, and returns the state this one ended in.
Carrying state is the whole point. A lexer restarted per line colors the
second half of a raw string as code.
Emitters are the upper layer, and HTML is one consumer rather than the
interface. Code writes a whole file. Diff writes a unified-diff
patch, where each row also carries its diff class.
Both emitters HTML-escape every byte they write. In cibot the source is a file a stranger pushed, so nothing in it can become markup.
Line orientation
A diff is the reason for it. A patch is a set of hunks with gaps, so there is no whole file to lex. Each row is colored on its own, given the state the row above it left, and rows the patch skipped leave that state approximate. That is acceptable in a diff, where the reader can see the gap, and exact for a whole file, where no line is missing.