Reproducible builds for a library of log format strings

An alert keyed to a log message breaks quietly when that message gets a different identity tomorrow. 10x takes that identity from a vocabulary of format strings read out of the source code. This post walks the compiler behind that vocabulary, stage by stage. Compile the same code twice and the vocabulary comes out the same, which is the guarantee the design is built around.

Every log line is printed by a format string in source code. log.info("User {} logged in", userId) sits in some file of some repo. Everything it prints except the user ID was fixed the day that line was committed. Collect the fixed parts across a codebase, the format strings, class names and level labels. That is the code's vocabulary.

At runtime the 10x Engine, a log-processing sidecar that runs beside the log forwarder (Otel, Fluent Bit), tokenizes every line and matches each token against that vocabulary. The match splits fixed text from values, so lines that differ only in their values are one message shape with one identity: a hash taken over the shape's template, the fixed text with a slot where each value goes.

Alerts and dashboards key on that identity. Fire when this error message spikes, compare this week's count of it to last week's. Both break quietly if the same message gets a different identity tomorrow. What follows is the compiler that reads the vocabulary out of the source code, plus the design calls behind one guarantee: compile the same code twice and the vocabulary comes out the same, entry for entry, in the same order.

The shape is a C build

Four stages. Pull fetches inputs from wherever code is stored. Scan turns each input into a symbol unit, one .10x.json per source file listing that file's printable vocabulary with a hash and a source line per entry, the .o object files of this build. Link merges the units into one library the way ld merges .o files into a .so. Push publishes the result.

What each stage does is spelled out in YAML pipeline definitions, one directory per stage and one per scanner, MIT licensed at github.com/log-10x/config and github.com/log-10x/modules alongside the grammars for the ANTLR parser generator and the default-library project list. The Java engine that executes those definitions is the closed, commercial half.

Scan already has a post of its own. Two stages produce the guarantee: the cache inside pull and the sort inside link.

The cache inside pull

Pull is one adapter per place code is stored: github clones repos, docker unpacks images, helm renders charts, artifactory fetches jars. The pull worth explaining is the one that re-ingests a previous run's symbol units, because it supplies the cache.

Every symbol unit records the 64-bit xxHash of the source file it was scanned from. Before scanning anything the pipeline hashes the file and looks for an existing unit carrying that checksum. A hit skips the scan and the old unit passes through to the linker untouched. Touch one file in a large repo and the next compile scans one file, everything else a checksum lookup. That hash is a cache key over the source tree and nothing more, not to be confused with the hash that identifies a message shape at runtime, which is taken over the shape's template.

How the scanners read code

A scanner collects the strings a file can print, with their source locations, then writes them into the file's symbol unit. One ANTLR tree-walker reads C and C++, C#, Go, Java, JavaScript, TypeScript, Python, Rust and Scala, each driven by one YAML file mapping grammar rules to capture modes, covered in the scanner post. The community grammar directory carries ready grammars for dozens of languages. Adding another is one grammar plus one more YAML file. Ruby, Kotlin, PHP, Swift, Lua, Groovy and bash get quoted-string extraction only, and compiled Java classes go through a bytecode scanner.

Three of them also carry a built-in native scanner that takes precedence unless disabled: JavaParser for Java (an open-source Java AST library), Python's own ast module and scalameta for Scala. The ANTLR walker came first, and on Java at production scale a dedicated parser finishes far sooner. The scanner post carries the timings behind that.

Native scanners read each file on its own. None of them follows an import or works out which class a name refers to: JavaParser decides a call is a logging call from the names of the methods and fields involved, then stops. Crude, though the vocabulary needs only the literals and their source locations, and no type solver changes either answer. Scanners can also run out-of-process, one subprocess per batch of files, results streamed back as JSON.

Ten scanners ship. The parsers above are antlr, javaParser, scalameta and pythonAST. Beyond them: executable runs external programs, described below; bytecode reads JVM class files; text reads structured text formats; pattern extracts by regular expression; archive unpacks compressed inputs and rescans the contents; symbol re-ingests previous units.

The executable scanner runs external programs and collects the candidate strings they print, so any process that prints candidates can feed the library through it, no grammar required. One recipe ships with it, running the operating system's strings command against binaries and keeping printable runs of four characters or longer, once a file check or the binary's own header has confirmed the input is one. A compiled binary stores its format strings as readable bytes, which is how that one recipe recovers vocabulary from artifacts no parser can read.

The sort inside link

Link takes every unit pull and scan produced and merges them into one set. Duplicates collapse: two copies of one unit with the same contents become a single copy. Then the sort rewrites the output with units in name order, units sharing a name ordered by their hash. Cache or fresh scan, whichever order the scan batches happened to finish in, the sort erases the difference. Each entry keeps the hash it had, since that hash is taken over the entry's content.

Every decision in the merge keys on content: the cache on file checksums, the sort on names and hashes. The sort can do nothing about a per-file scan timeout, though, and a loaded machine can drop a file that a fast one parses fully.

So the precise claim is a stable library given the same scan output.

What ships is a .10x.tar holding the merged .10x.json, a .10x.pb protobuf reverse index (the same entries keyed for the runtime's token matching) and a manifest. A tar header would ordinarily stamp the modification time of every file it packs, plus the user and group that packed it, so two runs producing identical symbols would write different bytes. The packaging step normalizes those fields. A test named sameInputsAtDifferentWallClocks_produceByteIdenticalTars builds two tars at different wall clocks and asserts they hash identically, and the headers leak no uid, gid or umask.

The lazier merge, appending units in whatever order they arrived, would have shipped faster and looked identical in every demo. Its failure mode is a library whose contents move when no code moved, which every downstream consumer reads as a change worth chasing. With the sort in place, and scan timeouts aside, a diff in the library points back at a diff in the source.

Publishing the compiled library

Push uploads the compiled symbol files to a target GitHub repo as a pull request that merges automatically, with no approval gate. The PR format still earns its place. The vocabulary lands as a commit someone can audit and revert after the fact. The sort holds the symbol content in a fixed order, so the diff shows the entries that changed and nothing else.

Two consumers read the merged result. The next compile run pulls the previous units back in, closing the incremental loop, with the cache skipping their unchanged sources so only new work reaches a scanner. The engine pulls the compiled vocabulary from the repo at startup, which makes whatever the repo contains the vocabulary a fleet runs.

The default library

Nobody compiles libc to use a C toolchain, and the same decision applies here. The runtime ships with a precompiled default library, built by running this same pipeline over public code. The project list is one YAML file in the open pipeline definitions, and anyone can add to it. That list names 126 repositories, Tomcat, Netty, Elasticsearch, Django, Gin, Envoy, Prometheus, Redis and Postgres among them. The shipped library carries a manifest recording every repository and container image its build read, each pinned to a commit or an image digest, so what went into it is checkable.

Compiling instead of clustering

The industry default is to infer structure at runtime: observe the stream, group lines that share a shape, and call each group a pattern. The grouping works, but the problem is the identity, because what a clustering parser calls pattern 47 depends on which lines it processed and in what order. We measured this on Drain, the dominant algorithm: two copies fed random halves of the same 197,430-line stream, then handed the same lines, disagree on the stored text for 39.78% of probed events whose shape occurs two to ten times.

Alerts and week-over-week volume comparisons keyed to those identities break when the identities change. Centralizing the clusterer across nodes narrows the gap, but even one shared instance cannot assign an identity before the first line arrives, and later refinement can re-split a group.

Traffic does not enter into the compile at all. The strings are already in a repo, a binary or a Helm chart before the first log line flows. The compiler reads them there and links them into one library whose identities are fixed by construction. A new pod changes nothing. A redeploy changes exactly the entries whose source changed, each traceable to an edit somebody reviewed.

Stream processors such as Cribl and Grepr transform logs in flight, and so does the 10x runtime. A transform pipeline has no equivalent of the half that runs before any log flows: a compiler that derives the vocabulary from source.

The reproducible-builds work settled this for compilers first: output that moves between runs for no reason cannot be trusted. A log vocabulary is a build output, so the same standard applies. The pipeline definitions are open at github.com/log-10x/modules, and the link stage definition is where to start reading.

Related: where log structure actually lives, one polyglot scanner and three native ones, and how a Drain pattern group changes as more lines arrive.