Cut the Datadog log bill per pattern

Datadog bills most for indexing, the charge that keeps log lines searchable. 10x reads each line against a vocabulary compiled from the application's own code, gives every logging statement a stable pattern ID, and tracks each ID's lines, GB, and cost per day. Two levers use that ID and keep every line: tier down routes a pattern to Flex Logs, Datadog's cheaper queryable tier, and offload routes a pattern's events to an S3 bucket the account owns, retrievable by ID. Both destinations are commodity; 10x adds the decision on which pattern is safe to move.

The expensive line on a Datadog log bill is indexing, the charge for keeping lines searchable; ingestion costs less. Datadog's own answer to that bill is Logging without Limits: ingest everything, index selectively, archive the rest to S3, rehydrate on the rare day an archived line is needed. The playbook keeps every line, and it does cut bills.

The playbook falls short somewhere else: its rules, written by hand or by Terraform, key on the text of the lines, stop matching when a deploy changes that text, and nothing re-makes the decision afterward. This post defends a different design: give every logging statement an ID that comes from the code rather than from the text of its output, track what each one costs, and move the ones that can safely leave the expensive tier, either down to a cheaper tier inside the platform or out to an S3 bucket the account owns. Nothing is deleted, and every line stays retrievable.

On Splunk, self-hosted Elasticsearch or OpenSearch, and ClickHouse, 10x compacts logs losslessly, a measured 63.7% on the public OpenTelemetry-demo sample counting the template dictionary, with a per-dataset spread of -4.6% to 96.1% across 14 LogHub sets, and each linked post shows the platform reading the compact form back at query time. Datadog cannot read that form, so compact is a no-op there; the two levers that work on Datadog are tier down and offload. The 10x engine runs as a sidecar to the log forwarder, the Datadog Agent included by way of Fluent Bit, and reads every line before Datadog receives it.

The 10x engine gives each line a stable pattern ID

Here are two lines from an API gateway.

2026-07-12 14:03:11 INFO api-gateway request complete route=/v1/charge status=200 bytes=1420 duration_ms=183
2026-07-12 14:03:12 INFO api-gateway request complete route=/v1/refund status=200 bytes=644 duration_ms=97

Both come from one logging statement in the code; only the values differ. The engine keeps the run of tokens that traces back to the statement itself and drops the values around it; that kept part is the pattern, and its hash is the pattern ID. The tracing is not guessed from the text: the engine carries a vocabulary of message tokens compiled from application and framework code, which ties a token to the statement that printed it. A default library compiled from common frameworks ships with the engine, and an account can compile its own code in the same way.

The ID is computed from the kept part alone, so the same statement maps to the same ID next week and on every host. The exception is a line with no code origin to key on, a raw access log for instance, where the kept run can include a variable token and one statement yields a few IDs instead of one.

From there the accounting runs per pattern. Datadog prices indexing per million events and ingestion per GB, so each ID accumulates both counts: lines per day, GB per day, and what that volume costs at the platform's rates. The budgets, the moves, and the audit below all read that series, one pattern at a time.

Here is one row of that accounting, measured over a recent week in a demo environment running the OpenTelemetry demo shop and priced at Datadog list rates. The pattern is a chatty INFO line from the cart service, ValkeyCartStore GetCartAsync called with userId, ID FU1__vh8hbY; the ID has held for the 30 days the metric window covers. Each event is a structured record carrying its resource attributes, about 1.5 KB.

The index line is 94% of what this one message costs, which is the whole case for a lever that targets indexing. The traffic is synthetic and the rates are list rates, so the dollars illustrate the accounting; a real account's negotiated rates differ, and no savings claim rests on this table.

Log Patterns finds the same shapes but keeps no durable ID

Datadog groups logs too. The Log Patterns view clusters the lines in the Log Explorer by shape, ranks the clusters by volume, and writes an exclusion filter from a cluster in one click. For a human who re-runs the search and recognizes the groups by eye, that view answers the question.

The clusters are computed per query, over the time range on screen, and no durable object stands behind them. An ID derived from clustering moves with the history that produced it, measured in its own post against Drain, the algorithm behind most pattern views. The exclusion filter the view writes is a match string written once: when a deploy changes the format string behind a shape, the filter stops matching, the volume returns to the indexed tier, and nothing attributes the creep back to the filter that stopped matching. Datadog also keeps no cost series per cluster; usage metrics slice by service and index, so what one cluster cost last quarter is not a recorded number.

The pattern ID is the durable version of that cluster. The grouping runs in the sidecar on every line, before Datadog receives it, so the ID does not depend on a query, a time range, or a host. A reformat around the message does not break it either: fields added around it or values bleeding into the text make a new variant of the line, but the message tokens the statement prints are unchanged, and they alone carry the identity, so the cost series runs through the deploy that silently breaks the exclusion filter. Changing the message's own words does start a new ID, and the new ID appears in the same accounting.

Tier down routes a pattern to Flex Logs and the lines stay queryable

Datadog sells Flex Logs, a cheaper tier inside the platform. A pattern routed there stops paying the standard index rate, the dominant line on the bill; ingestion is billed as before, and Flex bills its own lower rate for storage plus compute for the queries that read it. The lines stay queryable in Datadog's own Log Explorer, with no rehydration step and no second tool.

The routing is set once. The engine marks the events that should move with an attribute, and a Flex-only index whose filter matches that marker, ordered before the catch-all index, receives them; Datadog indexes are first match wins. Datadog does the enforcement with its own index filters, and nothing changes at the application.

The tier comes with limits. Flex queries run on Flex compute and are slower than the standard index, and log monitors evaluate indexed logs, so a pattern that feeds a monitor is exempted from moves entirely, on the protection list described in the decision section.

The default cost playbook starts with an archive: ship the low-value logs to S3, rehydrate the day they are needed. On Datadog I rank the levers the other way, and the reason is the query path. A pattern in Flex is still in the Log Explorer when an incident query needs it, and nobody waits on a rehydration job in the middle of an incident. The ranking holds on any platform whose cheap tier keeps lines queryable where the on-call already looks.

Offload moves a pattern's events to an S3 bucket the account owns

The second lever routes a pattern's events out of the platform entirely, and those events stop paying ingestion as well, because Datadog never receives them. The forwarder ships them to an S3 bucket owned by the account that owns the logs, as the forwarder's own JSON lines, readable with any S3 client; an index keyed by pattern ID is written alongside the objects. The index is a Bloom filter, the standard structure for ruling out files that cannot contain what a query asks for; keyed by pattern ID, it lets a retrieval for one pattern skip the objects that cannot contain that pattern's events. The index design, the filters stored in the S3 object keys themselves so one LIST over a time window streams them back, is its own post.

Getting events back never touches the platform. A retrieval by pattern ID reads the matching objects from the bucket and returns the events with nothing re-ingested into Datadog. An operator holding a raw line instead of an ID pastes the line into the same tooling, which resolves it to its pattern before the retrieval. Retention in the bucket is priced as S3 storage under the account's own policies.

Choosing which pattern is safe to move

Neither lever is novel as routing. Datadog already sells the cheap tier, and any forwarder can write to S3. Anyone can route logs somewhere cheaper; knowing which pattern is safe to move is the part the platform leaves open.

Cost alone does not make that call, and nothing moves by default. An operator sets a monthly dollar budget, the per-container byte budgets derived from it ship as a pull request a person merges, and a container with no merged budget is never regulated. The container is the unit the account already runs, named by a configurable field, the Kubernetes container name by default; the name is stable across replicas, so ten pods of one service fall under one policy.

Inside a budgeted container, a regulator in the sidecar decides which events move. The budget resolves to a byte cap per pattern per window, a share guard keeps the move on patterns that dominate the container's recent volume, and the overflow above the cap routes while every event at or below it stays in the standard tier. The window is a few minutes long, so a pattern that stops flooding is released on its own.

A move takes one of two grains: an operator names a pattern in a committed file and routes it outright, or the budget does it per window and only the overflow routes. In both, the pattern's first events of each window stay in the standard tier, and severity floors hold, with at least half of its Error events staying there by default. Which lever a pattern gets, Flex or the bucket, is part of the same committed entry: Flex for patterns still queried in the platform, the bucket for patterns that need no platform query at all. A wrong call keeps its data: every routed line is queryable in Flex or retrievable from the bucket by ID.

What a wrong call does change is counts. A monitor counting the indexed stream sees only the kept share, and offload is harsher: the platform never receives the events, so a log-based metric starves too. A Flex move leaves that metric whole, because Datadog computes it at ingestion.

The guards follow those lines. An operator exempts a pattern that feeds a monitor from moves, and one that feeds only a metric from offload; a steadily dominant request-complete pattern under a latency dashboard is exactly such an entry, and the regulator exists for the heartbeat and the retry loop, the patterns that flood with no dashboard reading them. The engine also publishes every pattern's pre-move volume as a metric series, with Datadog among the backends, so the volume series stays on a dashboard while the raw stream is split.

The exemptions and the budgets are the same committed files, each entry keyed by the pattern ID and carrying a reason and an expiry date, so a guard does not rot under the reformats that break a match string. Together with the per-pattern series they are the audit trail, the record a move is defended with later; volume is tallied per pattern before and after regulation, and the saving is measured against what was seen. None of it needs a pipeline change to try: the log10x MCP server installs into an AI client and boots in demo mode against a live demo environment, no account needed, and the table above is read from exactly that environment.

None of this asks Datadog to change anything. The tiers, the prices, and the enforcement are the platform's own; the decision comes from outside, from the sidecar that reads every line before the platform receives it. 10x decides, the platforms enforce.