Pattern ID stability and lossless expansion on 16 log datasets

Template miners group repeated log lines into templates, the fixed text with variables removed, and each template's pattern ID keys first-seen lookups and per-event graphs. Drain3, a widely used open-source miner, learns templates online, so a line's ID depends on what arrived alongside it. The 10x engine derives a content-addressed hash from each line's structure and keeps every value, typed, beside it. Across 16 public datasets, Drain3 at worst keeps the template for 0.8% of lines once other logs are appended; the 10x hash is identical in every arrangement on all 16, and every line expands back byte-for-byte.

Ask two ordinary questions about one message type in your logs: when did this event first appear, and what has its volume done since? Both are queries against a per-pattern time series, and both assume the pattern's ID is the same object in every file, on every host, in every window. If the ID moves, "first appeared" is an artifact of whichever instance you asked, and the graph is shards that do not join.

We measured that assumption on 16 public datasets, against Drain3, among the most widely used open-source template miners (hundreds of thousands of PyPI downloads a month). Take one of them, split it in two, and run Drain3 on one half by itself, then on that same half with the other half added after it. On the worst of the 16, fewer than 1 in 100 of those lines keeps the same template; the pattern ID moved for the other 99.2% on nothing but the log lines that arrived alongside them. The 10x engine assigns every line a content-addressed hash that is identical in every arrangement, on all 16 datasets: same line, same ID, any file, any order.

Stability is the first of two properties measured here, with identical raw bytes into both tools. The second is reversibility: Drain3 mines templates and discards the events behind them, while the 10x engine keeps every value, typed, beside the hash and can decode the exact original back, byte for byte.

Everything below regenerates from one public harness with pinned versions; every command is in the post.

Same raw bytes into both tools

loghub is a widely cited collection of labelled log datasets. We use the *_2k.log samples: 16 datasets (HDFS, Hadoop, Spark, BGL, Linux, Mac, Windows, Android, Thunderbird, OpenSSH, OpenStack, Apache, HPC, HealthApp, Proxifier, Zookeeper), each ~2,000 real log lines, ~32,000 lines total.

git clone --depth 1 https://github.com/logpai/loghub.git

Note: loghub commits these files with CRLF endings, and a checkout may hand you either CRLF or LF. Normalize to LF first, so both tools see byte-identical input and byte-exact comparisons mean the same thing on every platform:

import glob
for f in glob.glob("loghub/*/*_2k.log"):
    d = open(f, "rb").read()
    open(f, "wb").write(d.replace(b"\r\n", b"\n").replace(b"\r", b"\n"))

Fairness rule: identical raw input, no per-source configuration. Both tools get the whole raw line: timestamp, level, component, message. We do not hand-write a per-dataset regex to strip the header first, the way the standard loghub parsing-accuracy benchmark does. That is deliberate: the 10x engine's design goal is to ingest raw log lines with no per-source parsing rules to hand-write, so raw lines are the actual use case, not a handicap. The standard benchmark also measures a different thing, template accuracy against human labels rather than reconstruction. And header-stripping would not change the results below: Drain3's whitespace loss happens inside message bodies too, and its drift comes from order and context, not headers.

How we run Drain3

Drain3 is a Python library. We feed it raw lines with default config and read back the templates. It reconstructs nothing on its own, so to make a rebuild possible we recover each line's variables and refill the template. Drain groups its clusters by token count, so the variables sit at fixed token positions: align the line's tokens against the template's, take the line's token wherever the template has a <*>, and every value comes back with no regex to fail. This is the fair rebuild, and more than Drain3 ships with:

from drain3 import TemplateMiner
miner = TemplateMiner()

lines = open("loghub/HDFS/HDFS_2k.log", encoding="utf-8").read().splitlines()

# Pass 1: mine templates
for ln in lines:
    miner.add_log_message(ln)

# Pass 2: for each line, find its cluster and rebuild by token alignment.
# Drain clusters are length-homogeneous and <*> is always a whole token, so each
# value is the line's token wherever the template has <*>. This recovers every
# token; joining with single spaces is the one thing it cannot restore (see below).
def reconstruct(template, line):
    tt, lt = template.split(), line.split()
    if len(tt) != len(lt):
        return None
    return " ".join(lt[i] if tt[i] == "<*>" else tt[i] for i in range(len(tt)))

exact = 0
for ln in lines:
    m = miner.match(ln)                                   # final cluster for this line
    if m and reconstruct(m.get_template(), ln) == ln:
        exact += 1

print(f"reconstructable: {exact}/{len(lines)}")

How we run log10x

The 10x engine is a codec, run in two passes. The first, encode, reads a raw log file and writes two files: the template dictionary and the encoded stream. The config, tenx-encode.config.yaml:

tenx: run
include:
  - apps/shared                 # template assignment (the transform step)
  - run/modules/input/file      # read events from a file
  - run/modules/output/event/file

inputFile:
  - name: input
    path: $=TenXEnv.get("INPUT_FILE")
    printProgress: false

outputFile:
  # the template dictionary ("hidden classes"): {"templateHash","template"} per line
  - path: $=TenXEnv.get("OUTPUT_DIR") + "/templates.json"
    writeTemplates: true
    append: false
  # the encoded stream: ~<templateHash>,<value>,<value>,... per event
  - path: $=TenXEnv.get("OUTPUT_DIR") + "/encoded.log"
    fields: [ encode() ]

docker run --rm \
  -e OUTPUT_DIR=/out -e INPUT_FILE=/in/HDFS_2k.log \
  -v "$PWD/out:/out" \
  -v "$PWD/tenx-encode.config.yaml:/cfg/tenx-encode.config.yaml:ro" \
  -v "$PWD/loghub/HDFS:/in:ro" \
  log10x/pipeline-10x:1.1.5 "@/cfg/tenx-encode.config.yaml"

No API key is needed for local file I/O; the image runs under a built-in limited license that covers it. Each variable becomes a typed field in encoded.log; the template is a content-addressed hash of the line's structure. (The committed tenx-encode.config.yaml adds the standard enrichment chain, severity, multi-line grouping, message and HTTP fields; it is one generic config applied to all 16 datasets, not tuned per source, and it is what the scripts run to reproduce every figure below.) The second pass, decode, is the losslessness test further down: it takes these two files and rebuilds the original.

Does the same line get the same pattern ID?

Every "when did this event start?" is a first-seen lookup on a pattern ID. Every graph of one event's behavior over time is a counter keyed by that ID. Summing cost per message type across 100 hosts, deduping a noisy pattern fleet-wide, saying "pattern X grew 3× this week": all of it requires "pattern X" to be the same object every time it is computed, on every host, in every window. If the ID depends on what else was in the file, none of these questions has a stable answer.

Split a dataset into halves A and B, then run each tool on A, B, A+B, and B+A. For every line, record the pattern it gets (log10x's content-addressed hash; for Drain3 its final template string, since the numeric cluster IDs are per-run counters). Two questions: does a line's pattern survive reordering (A+B vs B+A), and does it survive other content being present (A alone vs inside A+B)?

log10x is 100% order-invariant and 100% context-invariant on all 16 datasets, and the set of pattern hashes for A+B always equals the set for B+A. That half is invariant by construction, a content-addressed hash is a pure function of the line, so the experiment's real content is the incumbent's behavior. Drain3 drifts. We expected order to be its weak spot, since an online learner that sees a different sequence should build different clusters. The data came back inverted: Drain3 is fully order-stable on eight of the 16 datasets, and where it collapses is context.

• Order drifts, moderately. In the Zookeeper logs, 16.6% of lines get a different template depending on whether the file is A+B or B+A: same bytes, different order (the Hadoop logs: 14.6%). In the Mac logs, the two orders even produce different template sets (308 templates one way, 311 the other; 37 exist only in one order, 40 only in the other).
• Context collapses. In the Windows logs, once the other half is appended after B, only 0.8% of B's lines keep the template they had when B ran alone. Almost nothing survives. The OpenSSH logs (40.3% of lines keep their template), Proxifier (23.4%), and Thunderbird (52.9%) show the same effect: the mere presence of other logs rewrites the patterns.

Drain3 is an online learner: it builds a prefix tree and generalizes a cluster's template as new messages arrive, so a line's final template depends on the sequence and set of everything seen. The 10x engine assigns a template from the structure of the individual line and identifies it by a content-addressed hash, a pure function of the line, independent of history. Same line, same hash, any file, any order. A first-seen computed on that hash is a property of the event, not of the instance that observed it, and a per-pattern counter keyed on it aggregates cleanly across hosts and windows. The values that differ from host to host, the hostname, PID, and timestamp, are pulled into typed fields and never enter the hash, which is why the same event type collapses to one ID across nodes; how that structural template is formed is its own post, Why every log line gets a stable identity. (A frozen, pre-trained Drain3 model is steadier for lines it already knows, but new lines still drift and it needs the training step; the 10x engine needs none.)

We measured the same Drain3 failure at 200 MB scale, against a production vendor's configuration, in "The Drain pattern ID is a join column you can't trust". That post ends by asserting the fix, a content-addressed hash, stable by construction. The numbers above are that assertion measured: head to head, per line, on all 16 datasets.

Getting the exact events back

A stable hash names the event type. The second property is that the encoded stream behind that hash still holds the events themselves, exactly, and this is what separates a codec from a miner. Drain3 mines templates and drops the values: out of the box it reconstructs nothing, because the variables were never stored. The 10x engine keeps every value, typed, beside the hash, so it can decode the exact original back. That is the decode pass. The encode step above wrote templates.json and encoded.log; concatenate them into one compact file and feed it back. The 10x engine needs no flag telling it these are encoded events: it auto-loads the template definitions, recognizes the ~-prefixed records, and writes their original text. The decode config, tenx-decode.config.yaml, is just an input and one output:

tenx: run
include:
  - apps/shared
  - run/modules/input/file
  - run/modules/output/event/file
inputFile:
  - name: input
    path: $=TenXEnv.get("INPUT_FILE")
    printProgress: false
outputFile:
  - path: $=TenXEnv.get("OUTPUT_DIR") + "/decoded.log"
    filter: isEncoded     # only the encoded records; write their reconstructed text
    fields: [ text ]

cat out/templates.json out/encoded.log > out/compact.log
docker run --rm \
  -e OUTPUT_DIR=/out -e INPUT_FILE=/out/compact.log \
  -v "$PWD/out:/out" \
  -v "$PWD/tenx-decode.config.yaml:/cfg/tenx-decode.config.yaml:ro" \
  log10x/pipeline-10x:1.1.5 "@/cfg/tenx-decode.config.yaml"
# out/decoded.log is byte-for-byte identical to the original HDFS_2k.log

The metric is per line: encode, decode, and count the lines that come back byte-for-byte identical. log10x reconstructs every one of the ~32,000 log lines, on all 16/16 datasets (100%), and the round trip holds at scale: on the full BGL dataset, 4,747,963 lines and 708.8 MB, every line reconstructs and the whole file is byte-identical.

In fairness, Drain3's content is recoverable too, and we are not claiming otherwise. Recovering it takes work that the 10x engine does out of the box. Drain3 stores no values, so its reconstruction rate is zero; the rebuild is code we bolted on. And that rebuild takes the original line as its input, re-tokenizing it to refill the template, so it only recovers a line still in hand. That is a tokenizer round-trip, not a store. With both, it recovers 100% of the content on all 16 datasets, every value, nothing garbled, losing only whitespace (a padded Jul 1 comes back as Jul 1). The 10x engine decodes the exact original from its stored files alone, no extra config needed. That is the real split: reconstruction from storage with no setup, against recovery that needs bolted-on code and the original log still in hand.

What the encoded record contains

Stability and reversibility are downstream of what the encoding actually is. One line from the HDFS logs and its encoded.log record:

raw:      081109 203615 148 INFO dfs.DataNode$PacketResponder: PacketResponder 1 for block blk_38865049064139660 terminating
encoded:  ~9PzqOgnWcl,081109,203615,148,DataNode,PacketResponder,1,38865049064139660
template: $ $ $ INFO dfs.$/$$: $1 $ for block blk_$ terminating   ($ marks a variable slot)

Every variable has been pulled into its own typed field: date, time, and PID from the header; the component; the responder id; the block id. The record is the template hash followed by those values. Drain3's equivalent is the template string with <*> holes; the values are not in its output at all unless you recover them by the alignment above, and even then they are opaque substrings, not typed columns. That is why the same pass that mines templates also gives you a reversible, queryable stream: drop, sample, aggregate, or tier a message type by its stable hash, and still decode the exact events back when you need them. Because every record is a line keyed by its hash, that retrieval is targeted: you pull back one specific event on demand, without expanding the rest of the stream.

Where Drain3 wins

The 10x engine does not come out ahead on every axis. One concession.

Drain3 produces fewer templates. Run cold, with no compiled symbol library, the 10x engine splits finely and produces more templates than Drain3 on 14 of 16 datasets (3,509 vs 1,761 across the suite; the content-only loghub ground truth is 1,363). The two exceptions cut the other way: in the HealthApp logs, Drain3 mints 656 templates to the 10x engine's 70. That is the price of a structure-preserving split: it will not merge two lines into one template unless they are genuinely the same shape. A finer split does not break aggregation: every one of those IDs is stable, so they roll up cleanly into coarser cost buckets and never move between windows, giving a per-message-type graph a longer tail to sum, not a shattered join. Symbol libraries, built by log10x's compiler from your source, collapse these further by recognizing fields semantically, but we did not measure that here.

Drain3's counts fall further, too, but only with per-source configuration. Give it masking rules (numbers, IPs, and hex IDs look like this) and HealthApp's 656 drops to 126, OpenSSH's 131 to 17, OpenStack's 46 to 22 (the masked figures are committed in facts.json). That masking is exactly the per-input tuning the 10x engine does without: you write and maintain a regex set for every log source, or the counts climb back. The engine's counts come from raw lines, out of the box.

Check the numbers yourself

If seeing the shapes in a log stream is the whole job, Drain3 in its default config is the right tool. The moment you key anything on a pattern, a first-seen, an alert, a graph of one event over time, a rule that drops or archives by message type, a stable ID stops being optional, and reversibility is what lets that rule discard bytes without losing events. That is what the 10x engine is built for, with no per-source configuration.

The full harness is public at github.com/log-10x/benchmarks: pinned versions, the exact configs above, the committed reference numbers (facts.json), and the scripts that regenerate every figure here, including two that re-derive the aggregates from the raw files rather than from the summary. Clone it and run it against loghub, or against your own logs.

Full per-dataset results

Three measurements per dataset. Templates: how many distinct patterns the tool ends up with. Lossless: the share of lines that expand back from the tool’s output byte-for-byte. Stability: the share of lines that keep their pattern when the same file runs in a different order or next to other data.

Template counts and log10x losslessness (both tools on identical raw lines):

Drain3 stores no values, so out of the box nothing comes back; the token-aligned rebuild we bolt on recovers 100% of the content but not exact whitespace, so its per-line byte-exact rate is a whitespace measurement rather than a data-loss one. The full per-dataset numbers are in the harness (facts.json).

Pattern stability is measured on the halves experiment from above: each cell is the percentage of lines that keep the same pattern between two runs. Reordered compares A+B against B+A, the same halves in the opposite order. Context A compares half A alone against the same half inside A+B; context B compares half B alone against the same half inside B+A.

For the 10x engine the table is uniform: 100% in every cell, all 16 datasets, all three comparisons. The same lines get the same IDs in either order, alone or together.

Drain3: