Cutting Elasticsearch log storage roughly in half without changing queries

Most of an Elasticsearch log index is the same format strings stored over and over. 10x stores each of those strings once and ships each event as a reference to it plus the values that change. A plugin at the Lucene layer keeps existing searches and Kibana working on that form. On a public sample the index measured roughly half its raw size.

Most of a log index is the same format strings stored millions of times. 10x encodes logs in the forwarding pipeline, before they reach Elasticsearch, shipping each format string once as a template and each event as the template's hash plus the values that changed. On a public 200 MB sample, that cut the on-disk Elasticsearch index from 28.6 MB to 13.8 MB, lossless.

L1ES is the plugin that makes existing text queries work unchanged over that form and puts the original line back before Kibana shows it. The plugin is MIT licensed at github.com/log-10x/elasticsearch-plugin. The 10x engine that writes the form is the paid product and is not in that repo. The templates sit in a plain Elasticsearch index, though, and standalone decoders reproduce every line from them.

The decoders that read the compact form back are MIT licensed and published on Maven Central: log10x-decoder-java, log10x-decoder-js and log10x-format. Only the engine that writes the compact form is the commercial product, so reading the data back needs no licence.

How an event is stored

One event from the OpenTelemetry demo's accounting service arrives in Elasticsearch like this:

{"stream":"stdout","log":"Accounting service started","docker":{"container_id":"80d793bf70a5dbe9408a516aa34542972497a8fdc0909b59c3c4957000c211a7"},"kubernetes":{"container_name":"accounting","namespace_name":"default","pod_name":"accounting-6f7cb8cd4d-hnqb4","container_image":"ghcr.io/open-telemetry/demo:2.1.3-accounting",...,"pod_ip":"192.168.33.161","host":"ip-192-168-42-205.ec2.internal",...}}

The raw event is 923 bytes, the envelope's middle trimmed for width.

Little in that line is unique to the event. The JSON keys and the fixed message words form its template, the full JSON skeleton with $ where values go, stored once in a separate template index. The data index stores the event as the template's hash plus its values:

~3;&4e;iuzQ,80d793bf70a5dbe9408a516aa34542972497a8fdc0909b59c3c4957000c211a7,accounting,6f7cb8cd4d,hnqb4,ghcr,2,1,3,3c3e...3418,308010eb,c129,-4320,a5ca,1ad8becbedda,192,168,33,161,-192,-168,-42,-205,accounting,accounting,...

The same event in compact form is 349 bytes, trimmed the same way. The leading ~ marks an encoded event, 3;&4e;iuzQ is the template hash, and everything after the first comma is the values, in template order.

The 10x engine derives the templates ahead of time from the source environment, so the hashes hold on every node sharing the pinned library and engine version, for the lines that library covers. A template learned from runtime data by tools like Drain is a running summary of what one instance saw, so the same line can carry different templates on different nodes. A hash of the template inherits that.

Why text search breaks

To Elasticsearch the two indices are unrelated, so a match query on the data index sees only hashes and values. Every word that came from a template's fixed text is missing there. A search for started returns nothing, and no mapping change fixes that.

Decoding every event at ingest restores search but erases the disk saving. A rewriting proxy in front of the cluster can edit query JSON, though it cannot reach into a Lucene segment from the REST API. So L1ES is a native plugin inside the Elasticsearch JVM, at the Lucene layer, where it can both rewrite the query and check a document inside its segment.

How the plugin answers a query

When a search arrives, `L1esQueryRewriter` walks the query tree and swaps each text query node (match, match_phrase, multi_match) for an L1ES-aware version, while compound nodes (bool, constant_score, boosting, dis_max) recurse and everything else passes through. Each rewritten node then sorts the candidate templates into three states. A query token is satisfied by a word in the template's fixed text or by a value in a $ slot. MUST_MATCH means the fixed text already carries every word the query asked for. CANNOT_MATCH means one of those words is missing from a template that has no $ slots to hold it. MIGHT_MATCH is the rest, where the missing word could turn up in a value.

Only MIGHT_MATCH documents pay a per-document check. L1esScorerWrapper wraps the inner Lucene scorer in a TwoPhaseIterator whose exact check decodes the candidate and tests the query against the original line. After scoring, L1esFetchSubPhase rebuilds the original line into _source, so Kibana Discover shows the log as printed.

The optional template-hash field

`L1esMatchQuery` first checks whether the data index carries a keyword field, l1x_tid, holding each document's template hash. With it, classification runs against an in-memory snapshot of the templates, with no Elasticsearch round-trip. Without it, the plugin queries the template index directly, which is slower but correct.

I could have made the field mandatory and deleted the slower path. I did not, because a plugin that demands a mapping change before it does anything is blocked until a reindex is scheduled. Any index that already holds encoded data can install the plugin and search today. Adding l1x_tid later enables the fast path on the next search, with no restart. The price is a second code path that has to stay correct.

What the plugin does not handle

Single-token queries are answered at the template level, so searching error returns every event of any template whose fixed text carries that word, even when the event is about something else. The extra hits are over-matches, adding events rather than altering or dropping them. Multi-token AND and phrase queries stay precise, because the per-document check verifies the decoded line.

Query types the rewriter does not cover (term, wildcard, regexp, and query_string, Kibana's Lucene syntax mode) see the encoded text directly, matching values but missing words that come from a template's fixed text, and so does anything else that reads the field, such as an aggregation or a runtime field. KQL compiles to the three rewritten query types, a field-less search becoming multi_match, so most Kibana usage takes the rewritten path. If the per-document check throws, the plugin keeps the document.

Relevance scores change, because a document matched through its template scores as a filter hit rather than BM25 over the original line. Log views sort by time, though. Highlighting runs against the stored encoded text without being rebuilt, so matched words are not marked in results. Decode work moves to query time as well, which costs more CPU than the same queries over raw text.

What the index measured

I indexed the 200 MB sample into Elasticsearch 8.17.0 twice, raw (one message text field per event) and compact, both with best_compression, one segment, and no replicas. A mapping that parses fields changes both sides and is not measured here. Each side keeps enough to reproduce every original line.

The raw index measured 28.6 MB across 197,430 documents. The compact pair measured 13.2 MB of data plus 0.6 MB of templates, 13.8 MB total, across 159,454 documents, because multi-line events merge into one during encoding. Decoding returned all 197,430 original lines byte-identical when the sample was round-tripped. The boundary sits one step earlier, at the file read, where bytes that are not valid UTF-8 are substituted before the engine tokenizes, the same substitution any UTF-8 log reader makes. The template index is part of the data. Cluster snapshots must include it, because events decode only against their templates.

Logsdb mode with synthetic _source on the same raw sample reclaimed under 4%, 28.7 MB to 27.7 MB. A block codec strips repetition only inside its window, while storing the template once takes it out of the data entirely.

The raw sample, the encoded form, and the templates are published in one release. Indexing the files reproduces the numbers. Every size in this post is on-disk index size.

Building for one version at a time

Elasticsearch loads a plugin only when it was built for the exact running version, and the descriptor pins it. Version coverage is therefore build-system work. One source tree targets an Elasticsearch or OpenSearch version through one Gradle property, the zip is named for the version it targets, and the OpenSearch variant is a subproject of the same tree that behaves identically, registered as a search pipeline. 8.17.0 and 2.19.0 are the versions the repo pins and tests, and neither ships as a released zip: the plugin is built from source.

Elasticsearch 9 and OpenSearch 3 moved to Lucene 10. The plugin requires Java 17 or later, and standalone Java and JavaScript decoders read the compact form outside Elasticsearch. That Lucene 10 port is a small, known change, three files touching APIs that moved, though it has not shipped yet.

Related: why the template hash stays stable, what else attaches to a template, where the symbol library comes from, the Splunk-side equivalent, and the same idea on ClickHouse in plain SQL.