Log compaction that stays searchable, measured on 15 public datasets

A log platform meters by the byte, and most bytes in a log line are text that the same statement prints every run. 10x stores that repeated text once and compacts the line before the platform meters it, and expand returns the original. Compact lines stay queryable in place rather than sitting in a gzip file. This post measures the reduction on fifteen public datasets, and the spread across them is wide.

A log platform (Splunk, for example) meters ingest by the byte, and most bytes in a log line are repetitive text. Part of it is what the code prints: the words of a format string, identical every time that statement runs. The rest is added around the message: the level, the service and pod names, the fields a logging framework stamps on every line (in Java, the MDC).

10x runs as a sidecar to the log forwarder, stores each line's repeated text once as a template, and ships each line as that template's hash plus the line's values.

Because each compact line is still a row the log platform can query, the reduction stays searchable in place instead of sitting in a gzip archive. The read path for Splunk, Elasticsearch, and ClickHouse is built in the sibling posts.

Here is one line from a Fluentd container, as the forwarder read it:

2025-10-02 00:17:22 +0000 [info]: #0 starting fluentd worker pid=18 ppid=6 worker=0

The engine stores the repeated part of that line once, as a template. Each $ marks a slot where a value goes:

{
  "templateHash": "9aZXwGwVG/",
  "template": "$(yyyy-MM-dd HH:mm:ss) +$ [info]: #0 starting fluentd worker pid=$ ppid=$ worker=$"
}

The line itself ships as a single row. After the leading ~ comes the template's hash, the same 9aZXwGwVG/ as in the dictionary entry above, and then that line's values in order: the timestamp as epoch milliseconds, the timezone digits, and the three numbers.

~9aZXwGwVG/,1759364242000,0000,18,6,0

The reverse step, expand, splices the values back into the template's slots and returns the original line, as the round-trip check further down measures.

This post measures the reduction on fifteen public datasets, run through the engine in its stock configuration: a 215 MB stream of Kubernetes container output, captured through Fluentd from a cluster running the OpenTelemetry demo, plus 14 of the 16 public LogHub datasets at full size.

The Docker image these runs used, log10x/pipeline-10x, is public, and so are all fifteen datasets, so every number below reruns with the one command at the end of the post.

Inputs and method

Each run is one pass of the engine over one dataset: the original file goes in, and two files come out, the encoded events and the template dictionary. The LogHub datasets are the full files from the project's Zenodo record rather than its 2,000-line benchmark excerpts. A full file reduces a few points more than the excerpt, because the one-time dictionary is spread over more events (HDFS: 26.4% on the excerpt, 29.1% at full size).

The two LogHub sets left out, Thunderbird (29.6 GB) and Windows (26 GB), are the collection's largest and get a separate run, so the 44.7% weighted figure below covers the 14 included sets and is not a LogHub-wide number.

The engine tokenizes each line and matches the tokens against a symbol library, the vocabulary of recurring tokens it recognizes. A library can be compiled from a specific codebase, and none of these runs did that. Every dataset went through the default symbol library that ships in the image. The default library is what a first install runs.

Reduction means the bytes of the compact form, the encoded events plus the template dictionary needed to expand them, against the bytes of the original file, with no gzip or ZSTD on either side.

The same stream measured three ways, each decoded byte-identical. As captured, metadata included: 64% (215,039,161 bytes in, 77,421,244 out, 3,028 templates). With the tag field our forwarder adds and the collector's debug output removed, metadata kept: 62.5% (88,975,911 in, 33,400,131 out). On the log message text alone, metadata stripped: 72.4% (40,551,755 in, 11,212,254 out). The first is the file as Fluentd shipped it. The second removes two things at once and is a sensitivity check, not a production figure. The collector's debug exporter is 47.6% of the lines and 50.8% of the bytes. Script.

What the runs measured

All fifteen datasets are below, sorted by reduction. Every row except OTel-demo is a LogHub set, and loghub-compaction reruns all fourteen with each one's counts.

The Events column counts events after multi-line grouping, a step in the stock configuration that joins runs of consecutive related raw lines into one event. That is how the 215 MB stream's 197,430 raw lines count as 157,228 events. No line is dropped. Expand returns the raw lines.

The 215 MB stream is the widest public data offers: 36 Kubernetes containers over about ten hours, the demo's own services (demo version 2.1.3) together with the cluster's infrastructure and system pods, every line carrying its Kubernetes metadata as Fluentd ships it. About half the bytes are the demo collector's debug exporter, which the demo enables by default. The method section reports the figure with and without it. The stream compacts 64%, a 2.78x ratio. The 14 LogHub datasets hold 52,276,243 events and 5.73 GB of input. Weighted by input bytes, they reduce 44.7%.

What drives the spread

The reduction tracks how much of each line is repeated text and how many lines share each template, not which application wrote it.

HealthApp (4.7%) and Android (20%) are the floor. Both are mobile logs whose component names and tags ship as values, beside the timestamps and IDs, so little of each line is repeated text. HealthApp's most common template, $(yyyyMMdd-H:m:s:SSS)|Step_$|$|$ $=$, keeps only |Step_ and the separators.

The middle of the table, 29% to 58%, sorts by the same variable. HDFS and OpenStack sit at the low end because their lines are mostly values: block IDs, PIDs, and byte counts, with little fixed text between them, and the values ship on every event regardless. Hadoop, Spark, and Apache sit at the high end because they write more repeated message text around their values. The same shape holds wherever a line is a row of values with thin separators. Access logs and VPC flow logs are in that class, so expect them near the low end.

Compact and compression

A general-purpose compressor can beat these ratios on log text. Compact is built for a different property: the reduced stream stays searchable, line by line, inside the platform.

A compact event sits in the platform as a row. A query filters on what is stored in the clear: the container, the template hash, the raw encoded text. Expand runs last, on the rows that survive the filter. Gzip applied to the same stream leaves a blob that answers no search until the whole archive is unzipped. That read path is built per platform, an Elasticsearch plugin at the Lucene layer or a ClickHouse view.

The other difference is position: compact runs in the sidecar, so the reduction takes effect before the platform meters ingest and before the indexer sizes its work. An at-rest codec runs after that, once the platform has already counted and indexed the original bytes. Running gzip on the transport changes nothing either, because platforms meter the log data itself and the transfer encoding is invisible to the bill.

That reduction reaches the bill where a platform prices by log volume. Splunk licenses by the volume ingested per day, so the reduction comes off the metered quantity directly. A self-managed store pays for the bytes it holds, but it also compresses what it holds, so the wire reduction does not carry to disk as-is. The disk number is whatever an in-place measurement says, and inside Elasticsearch that measurement came out at roughly half the on-disk size for typical log bodies. A platform that prices by event count works on a different axis, and compact is not the tool there.

Round-trip check

A reduction is only worth trusting if expand returns exactly what came in, so here is that check on two of the datasets: encode a file, concatenate the template dictionary and the encoded stream into one compact file, decode that file, and compare the result to the original. On both, expand returns the log text unchanged, with two exceptions: line endings come back as LF, and a file that does not end in a newline gets one.

The 215 MB OpenTelemetry stream returns all 197,430 lines byte-identical. The full Apache LogHub set, run the same way, returns all 56,482 lines identical.

The decode pass reproduces the original from the two files the encode pass wrote, with no key and no extra state, using the encode and decode configs published alongside the benchmark harness at github.com/log-10x/benchmarks:

cat <out>/templates.json <out>/encoded.log > <out>/compact.log
docker run --rm \
  -e INPUT_FILE=/out/compact.log \
  -e OUTPUT_DIR=/out \
  -v <out>:/out \
  -v <asset>/tenx-decode.config.yaml:/cfg/tenx-decode.config.yaml:ro \
  log10x/pipeline-10x:1.1.133 @/cfg/tenx-decode.config.yaml
# decoded.log matches the original, line for line

Reproducing a row

The datasets are public: the 215 MB stream is a capture of the OpenTelemetry demo's Kubernetes output, published in the log10x public-assets bucket, and the LogHub full datasets are on Zenodo. Each run is one container invocation.

docker run --rm \
  -e INPUT_FILE=/in/events.log \
  -e OUTPUT_DIR=/out \
  -v <out>:/out \
  -v <input>:/in/events.log:ro \
  -v <asset>/tenx-encode.config.yaml:/cfg/tenx-encode.config.yaml:ro \
  log10x/pipeline-10x:1.1.133 @/cfg/tenx-encode.config.yaml

Here <input> is the dataset file, <out> is a writable output directory, and <asset> is the directory holding the config, which is public at raw.githubusercontent.com/log-10x/benchmarks/main/loghub-compaction/tenx-encode.config.yaml. The pass writes the encoded events to encoded.log and the template dictionary to templates.json. Sum the two file sizes against the input and the reduction column reproduces.

Pull the image, run the command on a day of output from one cluster, and read the reduction from the two files it writes.