Extracting log templates from source, binaries, and Helm charts

Every log line is a format string plus runtime values, and 10x reads those strings from the code that prints them rather than guessing at ingest. A compiler runs pull, scan, link, push: it fetches repos, binaries, and Helm charts, scans each into symbol units, and links them into one library the runtime consults. A precompiled default built from 126 public repositories, Tomcat and Elasticsearch among them, means the compile step runs, for most stacks, on proprietary code alone; relinking the same inputs gives the same hashes, so dashboards hold across deploys that pin the same library and engine version.

A log line is a template plus some data, and the template does not have to be guessed at ingest. It sits in the code that emitted the line: your repos, open-source libraries, the compiled binary, the Helm chart, the jar in Artifactory.

We built a compiler that extracts the template from all of those and links them into one library, the way a C toolchain links .o files into a .so. The runtime then consults that library instead of inferring structure from the values it saw.

Here is the line that makes the case:

2025-09-29 09:01:23.456 INFO o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''

Most of this repeats across every Spring Boot restart. The class name TomcatWebServer, the format string Tomcat started on port(s): X (http) with context path Y, and the INFO severity are structural. The 8080 is data.

A vocabulary is the set of fixed strings the code can print: every class name, format-string literal, and severity label. The runtime checks each token against that set instead of guessing from how often it repeats.

Frequency guessing breaks on obvious cases. The same 8080 is a TCP port in one service and a row count in another:

Server listening on port 8080
Flushed 8080 rows to the segment store

A frequency guesser decides both 8080s are the same field and merges two unrelated shapes. The vocabulary has to come from a stable place: the code.

The compiler runs pull, scan, link, push

The compiler has the same shape as a C build.

• Pull. Fetch source files, binary artifacts, and Helm chart contents from their stores. Cache by content hash so unchanged inputs are never reprocessed.
• Scan. Run each input through the right scanner for its format. Output: per-file .10x.json symbol units, the .o of this build.
• Link. Merge the symbol units into one library, the way ld combines .o files into a .so.
• Push. Commit the library to a target GitHub repo. The runtime pulls from there at startup.

At runtime the vocabulary lets the engine blank the data from each line, leaving one template per shape. The hash is taken over that template, so identical shapes collapse to one for dedup and cost tracking.

Relink produces a stable library. The linker dedups the symbol units and writes them in a canonical order, sorted by unit name and then by unit hash, so the same inputs give the same symbol payload. Each shape's hash stays identical across deploys that pin the same library and engine version, for the lines that library covers, so dashboards and savings numbers do not reset every release (more on the runtime story). The tar wrapper around that payload is not bit-identical: its headers carry the build time, so the archive checksum moves even when the symbols do not.

Polyglot on two axes

A stack has variety on two axes: many source systems, many formats in each.

Most lines in a microservice come from Spring or the Kafka client, not your code. So the pull module carries an adapter per source: github, docker, helm, artifactory, plus a go.mod resolver that expands Go dependencies into GitHub pulls.

Inside any one source the content is heterogeneous. Source code spans seven-plus languages (Java, Go, Python, C#, C++, JavaScript, Scala), each routed to a scanner that knows its grammar: a dedicated parser for Java, Python, and Scala, and one shared ANTLR walker for the rest. Text configs carry structural strings as keys and labels. And then there are compiled binaries.

The executable scanner walks a binary and extracts printable strings, the same idea as the Unix strings command. Most log lines from a Go service live in the binary, not the source, so that scanner is not an edge case. It is also the extension point for any source that emits candidate strings.

Why we ship a precompiled default library

It comes down to overlap. Spring Boot is the same Spring Boot everywhere it runs, so recompiling the public projects everyone shares, once per install, is a tax on the common case.

So we ran the pipeline over public code and bundled the output. The project list is one YAML file in the open pipeline definitions: 126 repositories, Tomcat, Netty, Elasticsearch, Django, Gin, Envoy, Prometheus, Redis and Postgres among them. The shipped library carries a manifest naming every repository and image the build read, each pinned to a commit or a digest, so what went into it is checkable rather than asserted. The compile step then runs only for code the default does not cover, which for most stacks means the proprietary code alone.

Most tooling instead pushes the vocabulary problem onto you. Splunk sourcetype field extractions, Datadog log processing pipelines, Elasticsearch ingest grok rules, Cribl and Vector and Logstash transforms: prebuilt for the common integrations, hand-written for everything the application itself emits. Reading the vocabulary out of the code that prints the logs works only because the compiler reads every source and format in the environment, which is the cost of the approach and the reason it is not the common one.

The compile pipeline modules are open source at github.com/log-10x/modules/tree/main/pipelines/compile; clone it and run it against your own repos and images. The ANTLR scanner covers language support; hidden log classes covers what attaches to templates.