Cut the Elasticsearch log bill per pattern

An Elasticsearch cluster is sized by how much data sits in its hot tier, and a message printed in a loop adds to that every time it repeats. 10x gives each line a stable pattern ID, computed from the log statement that printed it rather than by clustering, and records each ID's volume. Two levers move a pattern and keep every line: tier down writes it to its own index, which an ILM policy mounts in the frozen tier, and offload sends it to a bucket the account owns. Deciding which pattern to move is what 10x adds.

The size of an Elasticsearch cluster follows how much data sits in its hot tier, because hot nodes carry the local SSD and the memory that keep that data searchable. Adding data there means adding nodes or a bigger deployment, so a message printed in a loop, a health check or a retry, grows what the cluster has to be sized for every time it repeats, whether or not anyone searches it.

10x shrinks the hot tier by moving the messages that take the most room out of it, and holding back the ones an alert or dashboard depends on. A large system's volume is dominated by a few messages that repeat, a health check, a retry, a request-complete line, printed millions of times each. Nothing is deleted: every line stays searchable, and the alerts and dashboards keep working. The hard part is choosing which messages are safe to move.

That choice needs a unit, and the engine's unit is a stable pattern ID, computed from the log statement that printed the line, so lines from one statement share an ID and each pattern carries its own volume. The engine reads every line before Elasticsearch receives it, running as a sidecar to your log forwarder (Fluent Bit, the OpenTelemetry Collector, Fluentd, and more). Two levers then move a chosen pattern, to a cheaper tier inside the cluster or to a bucket the account owns, and both keep the line. The levers cover the sources a forwarder fronts; logs an Elastic Agent integration ships straight into the cluster stay on whatever tier they land on.

The 10x engine gives each line a stable pattern ID

Here are two lines from an API gateway.

2026-07-12 14:03:11 INFO api-gateway request complete route=/v1/charge status=200 bytes=1420 duration_ms=183
2026-07-12 14:03:12 INFO api-gateway request complete route=/v1/refund status=200 bytes=644 duration_ms=97

Both come from one logging statement; only the values differ. Some tokens are the same in every line the statement prints: INFO, request complete, and the field names route, status, bytes. The rest change each time: the route, the status code, the byte count, the timestamp. The engine keeps the fixed tokens, the structure, and drops the changing values; what remains is the pattern, and its hash is the pattern ID.

The engine identifies the fixed tokens from a vocabulary, the strings the code can print, compiled from application and framework code ahead of time. It does not decide by how often a token repeats, and that is what makes the ID stable: the same statement gives the same ID next week and on every host. A line with no code behind it, a raw access log, is the exception, where a changing value can pass for a fixed token and one statement yields a few IDs.

Elasticsearch groups logs into shared shapes too, through the categorize_text aggregation that Kibana's log pattern analysis is built on, and through machine learning categorization jobs. Both run over documents that are already indexed, which is too late to route by: the index a line is written to is the tier that holds it, so the grouping has to exist before the line is written. The pattern ID does, because it comes from the vocabulary rather than the traffic. An ID derived from clustering also moves with the history that produced it, measured against Drain, a common clustering algorithm, in its own post. The pattern ID has no history to move with.

Each pattern has its own numbers. Here is one, measured over a week in a demo environment running the OpenTelemetry demo shop: a high-volume INFO line from the cart service, ValkeyCartStore GetCartAsync called with userId, ID FU1__vh8hbY, about 1.5 KB per event. It prints 16.2M lines a day, 24.2 GB, which is 486M lines and 726 GB over thirty days.

Every one of those gigabytes has to be held somewhere, and the index a line is written to is what decides which tier holds it. The traffic is synthetic, so the numbers show the accounting, not a saving.

Two levers that keep every line

Tier down moves a pattern to the frozen tier, where its data sits in a snapshot repository instead of on hot-tier SSD. The lines stay searchable from the same cluster with the same query, slower than hot because the shards read from the repository, and nothing leaves Elasticsearch. Self-managed that is disk the cluster stops needing; on Elastic Cloud Hosted, priced by the resources provisioned for it, the hot tier can shrink and the deployment has to be resized for that to land on the bill.

Offload sends a pattern's events out of the cluster, so they are never indexed. The forwarder writes them to an S3 bucket the account owns, as plain JSON lines any client can read, with a pattern-ID index beside them. A retrieval by pattern ID reads only the matching objects and returns the events, with nothing re-indexed. An operator who has a raw line instead of an ID pastes it into the same tool, which finds its pattern first.

A third lever applies where the cluster is the account's own to change. 10x can write the events in compact form, keeping each log statement's message text once as a template and shipping each line as that template's hash plus the values that change, with a plugin in the cluster reading that form back at query time. What it leaves on disk is measured in its own post, and the plugin it depends on rules it out on a managed deployment. Tier down and offload need nothing inside the cluster, which is what makes them the two levers that reach Elastic Cloud Hosted.

The forwarder picks the index and an ILM policy tiers it

On Datadog the down-tiered slice goes to a Flex index, on CloudWatch to an Infrequent Access log group, on Azure Monitor to a Basic or Auxiliary table, and on Coralogix to the same endpoint as everything else with a policy moving its class. Elasticsearch tiers by index, so the marked slice goes into an index of its own and an ILM policy on that index does the rest. There is no second endpoint to stand up and no plugin to install, which is what makes this the one lever that reaches a managed deployment.

The engine has to make the call, because it cannot be recovered from the event afterwards. Whether a pattern has passed its byte budget for the current window is a fact about a stream, counted across events in the sidecar, and a rule reading one document has nothing to compute it from. So the engine writes its decision onto the event and the forwarder reads it to pick the index.

{"@timestamp": "2026-07-12T14:03:11Z",
 "container": "checkout",
 "message": "request complete route=/v1/charge status=200",
 "tenx_hash": "FU1__vh8hbY",
 "routeState": "tier_down"}

routeState carries that decision and tenx_hash carries the pattern ID. Events marked tier_down go to one index and everything else to another, and only the first index has a policy that sends it to the frozen tier. Both fields are mapped as keyword, which is what keeps them searchable after the move.

That chain was run end to end on 2026-07-31 against a live Elastic Cloud Hosted deployment, version 9.4.4, with a second index kept hot as a control. The marked index reported 400 documents and zero bytes of local store on a frozen node, the control 200 documents and 19.2 kb on a hot node. Zero bytes is the saving: the documents are in the snapshot repository, and the control kept its own in hot, so only the marked slice moved.

Two things to know before reproducing it. Searchable snapshots are licensed, needing Enterprise self-managed or Gold and above on Cloud Hosted, plus a registered snapshot repository, which Elastic Cloud provides by default. ILM polls every ten minutes, so a transition is not visible the moment an index becomes eligible for one.

The pattern ID still resolves after the move

Cheap is worth nothing if the slice stops answering, so that was checked before anything was built. Both hashes were recorded before the transition and searched for after it. A term query on tenx_hash against the frozen index returned 400 of 400 documents, routeState was still aggregatable at 400 in the tier_down bucket, and the document bodies came back whole. The control answered 200 of 200 for its own hash from hot.

So the identity the sidecar stamps is what makes the moved slice findable later: an operator holding a pattern ID gets its events back with an ordinary search, from the same cluster, with no step in front of them. Had that query come back empty the lever would not have shipped, because a tier that costs less and answers nothing is worse than the hot index it replaced.

The decision is what 10x adds

Neither lever is new: Elastic sells the frozen tier, and any forwarder can write to a bucket. The open part is deciding which pattern is safe to move, and keeping that choice right as the logs change.

Nothing moves by default. An operator sets a monthly budget; the per-container limits from it are proposed by the log10x MCP in a pull request that a person merges, and a container with no budget is left alone. Inside a budgeted container, only a pattern that dominates recent volume, above a fifth of it over a few-minute window, has its excess moved.

Everything under that share stays hot, along with the first events of each window and, from severity floors, at least half of a pattern's Error events. A wrong choice loses no data: every moved line is searchable in the frozen index or retrievable from the bucket by ID.

The two levers carry different risk here. The frozen index stays inside the cluster, so a search or a dashboard that reads a down-tiered pattern still finds it, which is why tier down is the first lever on Elasticsearch and offload the fallback. Offload takes the events out of the cluster entirely, so anything reading them stops seeing them, and a pattern behind an alert is exempted from that lever. The engine also reports each pattern's pre-move volume as a metric series to the account's own metrics backend, so the volume stays on a dashboard either way.

The budgets and exemptions are committed files, keyed by pattern ID, each with a reason and an expiry, and with the per-pattern history they are the record behind every move. To see it on real data, the log10x MCP server installs into an AI client and runs against a live demo with no account; the table above came from there. 10x decides which pattern to move, and the platforms enforce it.