Measuring lossless log compaction on 15 public datasets

Each log statement prints the same fixed text every run; only the values between, the timestamps, IDs, and counts, change. 10x compacts logs before the platform meters them: a sidecar to the log forwarder stores the repeated text once as a template and ships each line as the template's hash plus values, and expand returns the original line. The compact rows stay queryable in Splunk, Elasticsearch, and ClickHouse rather than locked in an archive. Measured on fifteen public datasets, a 215 MB Kubernetes stream reduced 63.7%, and 14 full LogHub datasets 46.9% overall, from -4.6% to 96.1%.

A log platform (Splunk, for example) meters ingest by the byte, and most bytes in a log line are repetitive text. Part of it is what the code prints: the words of a format string, identical every time that statement runs. The rest is added around the message: the level, the service and pod names, the fields a logging framework stamps on every line (in Java, the MDC).

10x runs as a sidecar to the log forwarder, stores each line's repeated text once as a template, and ships each line as that template's hash plus the line's values.

Because each compact line is still a row the log platform can query, the reduction stays searchable in place instead of sitting in a gzip archive. The read path for Splunk, Elasticsearch, and ClickHouse is built in the sibling posts.

Here is one line from a Fluentd container, as the forwarder read it:

2025-10-02 00:17:22 +0000 [info]: #0 starting fluentd worker pid=18 ppid=6 worker=0

The engine stores the repeated part of that line once, as a template. Each $ marks a slot where a value goes:

{
  "templateHash": "9aZXwGwVG/",
  "template": "$(yyyy-MM-dd HH:mm:ss) +$ [info]: #0 starting fluentd worker pid=$ ppid=$ worker=$"
}

The line itself ships as a single row. After the leading ~ comes the template's hash, the same 9aZXwGwVG/ as in the dictionary entry above, and then that line's values in order: the timestamp as epoch milliseconds, the timezone digits, and the three numbers.

~9aZXwGwVG/,1759364242000,0000,18,6,0

The reverse step, expand, splices the values back into the template's slots and returns the original line, with the byte-for-byte guarantee and its one boundary measured further down.

This post measures the reduction on fifteen public datasets, run through the engine in its stock configuration: a 215 MB Kubernetes stream from the OpenTelemetry demo, plus 14 of the 16 public LogHub datasets at full size. The cause of the spread across them gets its own section.

The Docker image these runs used, log10x/pipeline-10x, is public, and so are all fifteen datasets, so every number below reruns with the one command at the end of the post.

Every number comes from the stock engine on public datasets

Each run is one pass of the engine over one dataset: the original file goes in, and two files come out, the encoded events and the template dictionary. The LogHub datasets are the full files from the project's Zenodo record rather than its 2,000-line benchmark excerpts; a full file reduces a few points more than the excerpt, because the one-time dictionary is spread over more events (HDFS: 26.4% on the excerpt, 29.1% at full size). The two LogHub sets left out, Thunderbird (29.6 GB) and Windows (26 GB), are the collection's largest and get a separate run, so the 46.9% weighted figure below covers the 14 included sets and is not a LogHub-wide number.

The engine tokenizes each line and matches the tokens against a symbol library, the vocabulary of recurring tokens it recognizes. A library can be compiled from a specific codebase, and none of these runs did that; every dataset went through the default symbol library that ships in the image. The default library is what a first install runs; a benchmark tuned per dataset is a number no first install reproduces.

Reduction means the bytes of the compact form, the encoded events plus the template dictionary needed to expand them, against the bytes of the original file, with no gzip or ZSTD on either side.

The mixed stream measures 63.7%; the LogHub datasets span -4.6% to 96.1%

All fifteen datasets are below, sorted by reduction, with nothing removed; every row except OTel-demo is a LogHub set. A negative reduction means the compact form came out larger than the input. Each dataset name links to the template dictionary that run produced, and the Encoded column links the compact event stream, so any row can be opened and checked against the exact bytes it came from. The three largest streams (Spark, HDFS, BGL) list a checksum in the manifest instead of hosting the encoded file.

The Events column counts events after multi-line grouping, a step in the stock configuration that joins runs of consecutive related raw lines into one event. That is how Android's 1,555,005 raw lines count as 72,999 events. No line is dropped; expand returns the raw lines.

The 215 MB stream is the widest public data offers: 36 services across 8 language runtimes, plus third-party infrastructure and the cluster's own system logs. It compacts 63.7%, a 2.75x ratio. The 14 LogHub datasets hold 50,310,461 events and 5.73 GB of input; weighted by input bytes, they reduce 46.9%.

The spread tracks how much of each line is repeated text

The reduction tracks how much of each line is repeated text and how many lines share each template, not which application wrote it. Both ends of the table follow from that one variable.

Android is the ceiling at 96.1%, a 25.39x ratio. Its 1,555,005 raw lines are framework and SDK messages in which a few shapes recur at enormous counts: the single most common line recurs 59,841 times, and another 40,941 times, with only timestamps and IDs varying between occurrences. Almost every byte on those lines is repeated text, so almost every byte factors out. That row stays out of the headline because a best-case outlier quoted as the expectation is a cherry-pick.

Proxifier (-4.6%), Mac (-2.9%), and HealthApp (-0.2%) are the floor. All three are small, high-diversity desktop and mobile logs where the distinct templates are a large fraction of the events; Mac resolves 6,745 templates from 23,736 events. On streams like these the one-time dictionary costs more than the per-event encoding saves, and the compact form comes out 0.2% to 4.6% larger than the input. A stream shaped like this is where compact is the wrong lever.

The middle of the table, 29% to 59%, sorts by the same variable. HDFS, Linux, and OpenStack sit at the low end because their lines are mostly values: block IDs, PIDs, and byte counts, with little fixed text between them, and the values ship on every event regardless. Hadoop, Spark, and Apache sit at the high end because they write more repeated message text around their values. The same shape holds wherever a line is a row of values with thin separators; access logs and VPC flow logs are in that class, so expect them near the low end.

Why not just gzip

A general-purpose compressor can beat these ratios on log text, and nothing in this post claims otherwise. Compact is built for a different property: the reduced stream stays searchable, line by line, inside the platform.

A compact event sits in the platform as a row. A query filters on what is stored in the clear, the container, the template hash, the raw encoded text, and expand runs last, on the rows that survive the filter. Gzip applied to the same stream leaves a blob that answers no search until the whole archive is unzipped. That read path is built per platform, an Elasticsearch plugin at the Lucene layer or a ClickHouse view, and the stored form does not make that work free.

The other difference is position. Compact runs in the sidecar, so the reduction takes effect before the platform meters ingest and before the indexer sizes its work. An at-rest codec runs after all of that, once the platform has already counted and indexed the original bytes. Running gzip on the transport changes nothing either, because platforms meter the log data itself and the transfer encoding is invisible to the bill.

That reduction reaches the bill where a platform prices by log volume. Splunk licenses by the volume ingested per day, so the reduction comes off the metered quantity directly. A self-managed store pays for the bytes it holds, but it also compresses what it holds, so the wire reduction does not carry to disk as-is; the disk number is whatever an in-place measurement says, and inside Elasticsearch that measurement came out at roughly half the on-disk size for typical log bodies. A platform that prices by event count works on a different axis, and compact is not the tool there.

Shrinking files on disk is work the storage engine already does; the metered side is where nothing was being removed, so a byte removed there is worth the most.

The compact form expands to the original bytes

A reduction is only worth trusting if expand returns exactly what came in, so here is that check on three of the datasets: encode a file, concatenate the template dictionary and the encoded stream into one compact file, decode that file, and compare the result to the original byte for byte, with line endings normalized to LF before the comparison.

The 215 MB OpenTelemetry stream returns all 197,430 lines byte-identical. The full Apache and Linux LogHub sets, run the same way, come back line for line, with one exception: a file that does not end in a newline comes back with one on its final line, and every byte before it holds.

So the guarantee is byte-for-byte on log text, which is what a platform stores and a query reads. The decode pass reproduces the original from the two files the encode pass wrote, with no key and no extra state, using the encode and decode configs published alongside the benchmark harness at github.com/log-10x/benchmarks:

cat <out>/templates.json <out>/encoded.log > <out>/compact.log
docker run --rm \
  -e INPUT_FILE=/out/compact.log \
  -e OUTPUT_DIR=/out \
  -v <out>:/out \
  -v <asset>/tenx-decode.config.yaml:/cfg/tenx-decode.config.yaml:ro \
  log10x/pipeline-10x:1.1.5 @/cfg/tenx-decode.config.yaml
# decoded.log matches the original, line for line

One command reproduces any row

The datasets are public: the 215 MB stream is a capture of the OpenTelemetry demo's Kubernetes output, published in the log10x public-assets bucket, and the LogHub full datasets are on Zenodo. Each run is one container invocation.

docker run --rm \
  -e LOG10X_MCP_INPUT_PATH=/mcp/input/events.log \
  -e LOG10X_MCP_OUTPUT_DIR=<out> \
  -v <out>:<out> \
  -v <asset>/tenx-mcp-file.config.yaml:/mcp/config/tenx-mcp-file.config.yaml:ro \
  -v <input>:/mcp/input/events.log:ro \
  log10x/pipeline-10x:1.1.5 @/mcp/config/tenx-mcp-file.config.yaml

Here <input> is the dataset file, <out> is a writable output directory, and <asset> is the directory holding the config, which is public at raw.githubusercontent.com/log-10x/log10x-mcp/main/assets/tenx-mcp-file.config.yaml. The pass writes the encoded events to encoded.log and the template dictionary to templates.json; sum the two file sizes against the input and the reduction column reproduces.

The output files behind every row are published too. A manifest lists each dataset's input, encoded, and dictionary byte counts with a SHA-256 for the encoded.log and templates.json it produced, and links those files, so a row can be checked against the exact bytes it came from without rerunning the engine.

The table stops at fifteen datasets, enough to establish the method and the spread but not the number for any stream it does not contain. Pull the image, run the command on a day of output from one cluster, and read the reduction from the two files it writes. That number is the answer for that stream, and it outranks every row above.