How a log line gets a stable pattern ID

Pattern grouping in most log platforms runs at query time, so one line can land in different clusters on different searches. 10x assigns each line a stable pattern ID instead: the engine labels each token as a source-derived symbol or a runtime value, drops the values, and hashes the symbols and delimiters into a 64-bit template hash. The same log format hashes to the same template on every node sharing the pinned library and engine version, for lines that library covers. On the 215 MB OTel-demo sample, 159,450 events collapse to 2,933 templates, a measured 63.7% reduction counting the dictionary.

Most pattern-detection in mature observability platforms runs at query time. Splunk's cluster command, Datadog's Log Patterns view, Elastic's categorize_text aggregation all work this way. Feed any of them the same log line through two different searches and you can land in two different clusters, under a grouping key that was rebuilt for that query. Fine for a dashboard, useless for tracking a pattern across queries.

Runtimes solved this for objects. V8 gives every object a hidden class, a stable descriptor of its shape, computed once and shared by every same-shaped object for the program's life. We built 10x to do the same for log events. Every event gets a template, the hidden class for a log line.

A stable ID matters when something downstream has to stay attached to the same pattern tomorrow, like an alert on one log shape or a per-pattern cost budget.

On the 215 MB OTel-demo sample, 159,450 events collapse to 2,933 templates, a measured 63.7% byte reduction counting its 3.2 MB template dictionary, and the same line hashes to the same template on every node sharing the pinned library and engine version, for the lines that library covers. The sample is public; the encoder that produces the hash is the commercial engine, so the determinism evidence a reader can check today is the measured contrast in the Drain post, not an encode-your-own-line run.

What a symbol is, and why structure isn't in the line

A log event's shape isn't declared anywhere. Two events from different log.info(...) calls are different shapes even when they look alike; two with the same skeleton and different runtime values are the same shape.

To tell those apart, the engine needs a symbol: a fixed string it knows came from source code, like a class name or a format string, not from runtime data. That split between structure and value is the whole trick.

The vocabulary of symbols comes from a library. 10x ships a default library built by scanning 126 open-source repositories and 14 public container images. An optional compile pass scans your repos and container images and adds your own code. (Where the symbol library actually comes from is its own post.)

How a log line becomes a stable hash

At runtime the engine tokenizes the line and labels each token as a known symbol or a variable value. It drops the variable values, then computes a 64-bit FarmHash over the symbols and the delimiters between them: the brackets, quotes, and punctuation that belong to the log format.

The hash input is only source-derived symbols, never runtime values. So the same log format produces the same template hash on every node sharing the pinned symbol library and engine version, across restarts and environments.

One line, two identities

That template hash names the exact shape of a line, and shape is finer than statement. One log.info call can print several shapes over its lifetime: a field gets appended, a wrapper adds a prefix, the timestamp format changes at a collector upgrade. Each of those is a different template with a different hash, and each still comes from one place in the code.

So the engine keeps a second identity above it. The pattern is a subset of the template: the run of symbols the engine traces back to a single source origin, kept and hashed on its own as tenx_hash. Several templates that trace to the same origin share one pattern, which makes a set of templates sit under a single name.

The two answer different questions. The template hash says which stored shape rebuilds this line, so it is the join key that makes compact storage work. The pattern says which statement in the code emitted it, so it is what a per-pattern budget, a routing rule, or a cost report keys on. A budget written against a template hash splits the moment the format shifts under it; written against the pattern, it holds through the shift.

One line from the OTel-demo sample shows the split. Grafana's alert sender writes this 623 times:

logger=ngalert.sender.router rule_uid=des78nlna99tsf org_id=1 t=2025-10-01T20:13:00.042088606Z level=info msg="Sending alerts to local notifier" count=1

Go writes fractional seconds without trailing zeros, so the same statement prints timestamps of four different widths, and each width is a different shape with its own template hash:

t=$(yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS'Z')   562 events
t=$(yyyy-MM-dd'T'HH:mm:ss.SSSSSSSS'Z')     55 events
t=$(yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z')       5 events
t=$(yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z')        1 event

Four templates, four hashes, all tracing to one line of Grafana, so all four sit under one pattern. A cost report keyed on the pattern counts all 623. Keyed on a template hash it counts 562 and misses the rest without saying so.

An example, on OTel-demo data

Here's a fluentd startup line from the OTel-demo sample that ships with 10x, abridged to its log message (the JSON envelope and kubernetes metadata get templated too):

2025-10-02 00:17:22 +0000 [info]: #0 starting fluentd worker pid=18 ppid=6 worker=0

The first time this shape appears, the engine extracts the template ($ marks each variable slot):

$(yyyy-MM-dd HH:mm:ss) +$ [info]: #0 starting $ worker pid=$ ppid=$ worker=$

Every later event becomes its template hash plus the values that filled the slots:

~R}>PZj;Jdp,1759364242000,0000,fluentd,18,6,0,...

The leading blob is the template hash. The rest is the timestamp, timezone, worker name, pid, ppid, and worker number, then the kubernetes and container metadata. Across the full 215MB OTel-demo log file, 159,450 events distill into 2,933 unique templates and 74.8MB of encoded events: a measured 63.7% byte reduction counting the 3.2MB template dictionary that travels with them.

The reduction is what gets quoted. The determinism is what the rest of the system is built on. (Log events have class knowledge covers what opens up once every event has a stable identity.)

A design choice that runs against instinct

The instinct is to weight toward the symbols that show up most often. I'd argue the opposite, and built the engine to argue it: the more places a symbol appears, the less useful it is for picking the right origin.

Take Logger. In any real Java application it appears in hundreds of files, so it narrows the candidates from "everything" to "everything." Take BillingReconciler, three places maybe. Seeing it narrows the search dramatically.

So we capped it. The engine's default maxSymbolUnitsPerToken is 128: the most source locations it will consider for any one symbol. We tried raising it later for accuracy. It doesn't help: the extra slots fill with exactly the high-frequency noise the cap was keeping out. I'd have a hard time making a stronger claim about anything else in the engine.

Where to look

The full 215 MB OTel-demo sample is published as a release asset on the open-source config repo, with an 8 MB gzip alongside. The templates and the encoded events the engine produces on that sample ship in the same release, so the counts above are countable from the assets (the repo's data/ directories are runtime placeholders). The full runtime API reference is at doc.log10x.com. For code that already holds encoded events, standalone Java and JavaScript decoders rebuild the original lines. The claim to hold against all of this: the same line hashes to the same template on every node sharing that pinned library and engine version, or the architecture is wrong.