Reproducible builds for a library of log format strings

Format strings in source code print every log line; collected across a codebase, those strings are its vocabulary. The 10x Engine, a log-processing sidecar, matches every line against it, so lines differing only in their values share one identity, a hash over the fixed text, which alerts and dashboards key on. This post covers the compiler behind the vocabulary, four stages shaped like a C build: a checksum cache skips unchanged files, and the linker sorts by name and hash, so the same code compiles to the same library, entry for entry, in the same order, scan timeouts aside.

Every log line is printed by a format string in source code. log.info("User {} logged in", userId) is a line in some file of some repo; everything it prints except the user ID was fixed the day that line was committed. Collect the fixed parts across a codebase, the format strings, class names, and level labels, and that is the code's vocabulary.

At runtime the 10x Engine, a log-processing sidecar that runs next to your log forwarder (e.g., Otel, Fluent Bit), tokenizes every line and matches each token against that vocabulary. The match splits fixed text from values, so lines that differ only in their values are one message shape with one identity: a hash taken over the shape's template, the fixed text with a slot where each value goes.

That identity is what alerts and dashboards key on: fire when this error message spikes, compare this week's count of it to last week's. Both quietly break if the same message gets a different identity tomorrow. This post is about the compiler that reads the vocabulary out of the source code and the design calls behind one guarantee: compile the same code twice and the vocabulary comes out the same, entry for entry, in the same order.

The shape is a C build

The compiler runs in four stages. Pull fetches inputs from wherever code is stored. Scan turns each input into a symbol unit, one .10x.json per source file, listing that file's printable vocabulary with a hash and source line per entry; the units are this build's .o object files. Link merges the units into one library the way ld merges .o files into a .so. Push publishes the result.

What each stage does is spelled out in YAML pipeline definitions, one directory per stage and one per scanner, Apache-2.0 at github.com/log-10x/config and github.com/log-10x/modules alongside the grammars for the ANTLR parser generator and the default-library project list. The Java engine that executes those definitions is the closed, commercial half.

Scan already has a post of its own. Two stages produce that guarantee: the cache inside pull and the sort inside link.

The second build skips everything that did not change

Pull, the first stage, is one adapter per place code is stored: github clones repos, docker unpacks images, helm renders charts, artifactory fetches jars. The pull worth explaining is the one that re-ingests a previous run's symbol units, because it supplies the cache.

Every symbol unit records the 64-bit xxHash of the source file it was scanned from. Before scanning any file, the pipeline hashes the file and looks for an existing unit carrying that checksum; a hit skips the scan entirely and the old unit passes through to the linker unchanged. Touch one file in a large repo and the next compile scans one file. Everything else is a checksum lookup. That hash is a cache key over the source tree and nothing else; the hash that identifies a message shape at runtime is a separate thing, taken over the shape's template.

How the scanners read code

A scanner collects the strings a file can print, with their source locations, and writes them into the file's symbol unit. This layer is also where the pipeline is extensible. One ANTLR tree-walker reads seven source languages, C++, C#, Go, Java, JavaScript, Python, and Scala, each driven by one YAML file mapping grammar rules to capture modes, covered in the scanner post; adding a language means taking a grammar from the community grammar directory, which has ready grammars for dozens of languages, and writing that one YAML file. Three of the seven also carry a built-in native scanner that takes precedence unless disabled: JavaParser for Java, an open-source Java AST library; Python's own ast module; scalameta for Scala. The ANTLR walker came first, and Java at production scale is what pushed those three to native parsers.

The native scanners read each file on its own; none of them follows an import or works out which class a name refers to. JavaParser decides a call is a logging call from the names of the methods and fields involved, and that is where it stops. For collecting candidate strings, names have been enough; the vocabulary needs the literals and their source locations, and no type solver changes either answer. Scanners can also run out-of-process, one subprocess per batch of files with results streamed back as JSON. Ten scanners ship in total: antlr, javaParser, scalameta, and pythonAST are the parsers above; executable runs external programs and is described below; bytecode reads JVM class files; text reads structured text formats; pattern extracts by regular expression; archive unpacks compressed archives and rescans the contents; symbol re-ingests previous units.

The executable scanner runs external programs and collects the candidate strings they print, so any process that prints candidates can feed the library through it, no grammar required. The recipe that ships runs the operating system's strings command against binaries, after a file check or the binary's own header confirms the input is one, and keeps printable runs of four characters or longer. A compiled binary stores its format strings as readable bytes, so that recipe alone recovers vocabulary from artifacts no parser can read.

The linker is what keeps the vocabulary stable

Link merges every unit that pull and scan produced into one set, which is what linking is. The merge collapses duplicates, two copies of one unit with the same contents, to a single copy; a duplicate adds no information to the copy that remains.

After the merge comes the sort. The output is rewritten with units in name order, and units that share a name in the order of their hash. Whether a unit came from the cache or a fresh scan, and whichever order the scan batches happened to finish in, the sort erases the difference. The merged symbol file comes out carrying the same units in the same order, and each entry keeps the hash it had, because that hash is taken over the entry's content.

The guarantee is mechanical, and it stops where the scan does. Every decision in the merge keys on content: the cache on file checksums, the sort on names and hashes. What the sort cannot fix is a per-file scan timeout, since a loaded machine can drop a file that a fast one parses fully. So the precise claim is a stable library given the same scan output.

The artifact is a .10x.tar containing the merged .10x.json, a .10x.pb protobuf reverse index (the same entries keyed for the runtime's token matching), and a manifest. That guarantee covers what the archive holds and not the archive itself. A tar header stores the modification time of every file it packs, plus the user and group that packed it, so two runs producing identical symbols still write different archive bytes. Normalizing those header fields is packaging work I have not done, which means hashing the .10x.tar is no way to tell whether the vocabulary changed. Comparing two libraries means comparing their contents.

The lazier merge, appending units in whatever order they arrived, would have shipped faster and looked identical in every demo. Its failure mode is a library whose contents change when no code changed, which every downstream consumer treats as a real change. With the sort in place, whoever consumes this library gets to assume that, scan timeouts aside, changed contents mean changed code.

The vocabulary ships as a pull request

Push uploads the compiled symbol files to a target GitHub repo as a pull request that merges automatically, with no approval gate. The PR format matters because the vocabulary lands as a commit someone can audit and revert after the fact, and because the sort keeps the symbol content in a fixed order, so a diff shows the entries that changed and nothing else.

Two consumers read the merged result. The next compile run pulls the previous units back in, closing the incremental loop: the reuse pull supplies the old units, the cache skips their unchanged sources, and only new work reaches a scanner. The runtime is the second consumer. The engine pulls the compiled vocabulary from the repo at startup, so the vocabulary a fleet runs is whatever the repo contains.

A default library covers the open-source stack

Nobody compiles libc to use a C toolchain, and the same decision applies here. The runtime ships with a precompiled default library, built by running this same pipeline over public code. The project list is one YAML file in the open pipeline definitions, and anyone can add to it; it names 126 repositories, Tomcat, Netty, Elasticsearch, Django, Gin, Envoy, Prometheus, Redis, and Postgres among them. The shipped library carries a manifest recording every repository and container image its build read, each pinned to a commit or an image digest, so what went into it is checkable. The only code that needs compiling locally is proprietary code; the vocabulary for those projects arrives with the engine.

Why not cluster the logs instead?

The industry default is to infer structure at runtime: observe the stream, group lines that share a shape, and call each group a pattern. The grouping works; the problem is the identity, because what a clustering parser calls pattern 47 depends on which lines it processed and in what order. We measured this on Drain, the dominant algorithm, using Jaccard overlap, the fraction of patterns two runs agree on: reorder one stream and the template set holds at 0.93; split the stream across instances, which is what running more than one node means, and overlap collapses to 0.18.

Alerts and week-over-week volume comparisons keyed to those identities break when the identities change. Centralizing the clusterer across nodes narrows the gap, but even one shared instance cannot assign an identity before the first line arrives, and later refinement can re-split a group.

The compiler does not depend on traffic at all. The strings are already in a repo, a binary, or a Helm chart before the first log line flows; the compiler reads them there and links them into an artifact whose identities are fixed by construction. A new pod changes nothing, and a redeploy changes exactly the entries whose source changed, each traceable to a source edit somebody reviewed.

Stream processors such as Cribl and Grepr transform logs in flight, and so does the 10x runtime. The difference is the half that runs before any log flows: a compiler that derives the vocabulary from source, which a transform pipeline does not have.

The argument is the reproducible-builds argument: output that differs between runs for no reason cannot be trusted, and a log vocabulary is a build output like any other. The pipeline definitions are open at github.com/log-10x/modules; the link stage definition is where to start.

Related: where log structure actually lives, one polyglot scanner and three native ones, and the Drain pattern ID you can't trust.