Cut the Datadog log bill per pattern

Indexing, the charge that keeps log lines searchable, is the expensive part of a Datadog bill. The usual fix is an exclusion rule written against the line's text, which stops matching when a deploy reformats the line. 10x gives each line a pattern ID compiled from code and run configuration, the same on every node, and tracks its cost. Patterns that can safely leave the expensive tier move to a cheaper tier or to an S3 bucket the account owns.

Indexing is what makes a Datadog log bill expensive. Keeping lines searchable is what indexing pays for, which costs more than ingesting them in the first place. Datadog answers that bill with Logging without Limits, a playbook that ingests everything, indexes selectively, sends the rest to S3 and rehydrates it on the rare day an old line is needed. Bills do come down that way, with every line kept.

Where the playbook falls short is the rules, though. They key on the text of the lines, so a deploy that adds a field or reformats a value stops the rule matching, and nothing re-makes the decision afterward. Each line gets a pattern ID compiled from code and the run configuration, the same on every node, which survives deploys that change values and formatting around a statement. Every ID carries what it costs, and the patterns that can safely leave the expensive tier move down to a cheaper tier inside the platform or out to an S3 bucket the account owns.

A stable pattern ID

Here are two lines from an API gateway.

2026-07-12 14:03:11 INFO api-gateway request complete route=/v1/charge status=200 bytes=1420 duration_ms=183
2026-07-12 14:03:12 INFO api-gateway request complete route=/v1/refund status=200 bytes=644 duration_ms=97

Both come from one logging statement in the code. Only the values differ. The engine keeps the run of tokens that traces back to the most likely statement and drops the values around it. That kept part is the pattern. Its hash is the pattern ID. That tracing runs off a vocabulary of message tokens compiled from application and framework code, which ties each one to the code that holds it. A default library compiled from common frameworks ships with the engine. An account can compile its own code the same way.

The ID is computed from the kept part alone, so the same line maps to the same ID on every node running the same engine, library and configuration.

From there the accounting runs per pattern. Datadog prices indexing per million events and ingestion per GB, so each ID accumulates lines per day, GB per day, and what that volume costs at the platform's rates. Budgets, moves, and the audit further down all read that series, one pattern at a time.

Here is one row of that accounting, measured over a recent week in a demo environment running the OpenTelemetry demo shop and priced at Datadog list rates. The pattern is a chatty INFO line from the cart service, GetCartAsync_called_with_userId, ID YZplfQ4a1Xw. Each event is a structured record carrying its resource attributes, about 1.5 KB.

Indexing carries 94% of what this one message costs, which is the whole case for a lever that targets it. The traffic is synthetic, though, and the rates are list rates, so the dollars illustrate the accounting. No savings claim rests on this table, since negotiated rates differ.

Datadog Log Patterns

Datadog groups logs too, in a view called Log Patterns. That view clusters the lines in the Log Explorer by shape, ranks the clusters by volume and writes an exclusion filter from one of them in a click, which does answer the question for anyone who re-runs the search and recognizes the groups by eye.

Clusters are computed per query over the time range on screen, though, so nothing durable stands behind them, and an ID derived that way moves with the history that produced it, which a separate post measures against Drain. The exclusion filter has the same weakness. A deploy changes the format string, the match string stops matching, the volume returns to the indexed tier, and no record ties that creep back to the filter. No cost series per cluster is kept either, so what one cluster cost last quarter was never recorded.

The pattern ID is the durable version of that cluster. Grouping runs in a sidecar to the log forwarder, which can be the Datadog Agent by way of Fluent Bit, and reads every line before Datadog receives it, so the ID does not depend on a query, a time range or a host. Fields added around the message, or values bleeding into the text, make a new variant of the line and leave the ID alone, because only library words carry the identity, and a value enters it only when it is itself one. Changing the words of the message does start a new ID, though, and that ID appears in the same accounting.

Tier down to Flex Logs

Inside the platform Datadog sells a cheaper tier called Flex Logs. A pattern routed there stops paying the standard index rate, which is the dominant line on the bill. The lines stay queryable in Datadog's own Log Explorer, with no rehydration step. Ingestion is billed as before, though, and Flex charges its own lower rate for storage plus compute for the queries that read it.

The routing is set once. Datadog does the enforcement with its own index filters, while the application keeps logging as it did. The engine marks the events that should move with an attribute, and a Flex-only index whose filter matches that marker receives them, because Datadog indexes are first match wins and that index is ordered before the catch-all.

Flex comes with limits. Log monitors evaluate indexed logs, so a pattern feeding one is exempted from moves entirely. Queries there run on Flex compute, though, and are slower than the standard index.

The default cost playbook starts by shipping the low-value logs to S3 and rehydrating them the day they are needed. On Datadog I rank the levers the other way, and the reason is the query path. A pattern in Flex is still in the Log Explorer when a query needs it, and nobody waits on a rehydration job in the middle of an incident.

Offload to an S3 bucket

Offload, the second lever, routes a pattern's events out of the platform. Those events never reach Datadog, so the account stops paying ingestion as well as indexing. The forwarder writes them to an S3 bucket the account owns, in its own JSON lines. Any S3 client can read them.

An index is written alongside those objects and keyed by pattern ID, and being a Bloom filter it lets a retrieval read only the ones that might carry the pattern it asks for and skip the rest. The filters themselves live in the S3 object keys, which one LIST over a time window streams back, though that design is its own post.

A retrieval by pattern ID reads the matching objects from the bucket and returns the events with nothing re-ingested into Datadog. An operator holding a raw line instead of an ID pastes the line into the same tooling, which resolves it to its pattern before the retrieval. Retention in the bucket is priced as S3 storage under the account's own policies.

Choosing which pattern moves

Datadog already sells the cheap tier and any forwarder can write to S3, so the open part is knowing which pattern is safe to move.

Cost alone does not make that call. Nothing moves by default. Setting a monthly dollar figure comes first, and the per-container byte budgets derived from it ship as a pull request a person merges. A container with no merged budget is never regulated. Containers are the unit because the account already runs them and their names hold across replicas, so ten pods of one service fall under one policy.

Inside a budgeted container, a regulator in the sidecar decides which events move. The budget resolves to a byte cap per pattern per window, a share guard keeps the move on patterns that dominate recent volume, and only the overflow above the cap routes while everything at or below it stays in the standard tier. The window runs a few minutes, so a pattern that stops flooding is released on its own.

A pattern's first events of each window stay in the standard tier, and severity floors keep at least half of its Error events there by default.

The lever is set per container in a committed file, and every pattern over its cap in that container takes it: Flex where the container's lines are still queried in the platform, the bucket where they need no platform query at all. A wrong call keeps its data, because every routed line is queryable in Flex or retrievable from the bucket by ID.

What a wrong call does change is counts. A monitor counting the indexed stream sees only the kept share. Offload is harsher, because the platform never receives the events, so a log-based metric starves too. A Flex move leaves that metric whole, though, since Datadog computes it at ingestion.

Patterns that feed a monitor are exempted from moves, and so are ones that feed a log-based metric in a container whose lever is offload. A steadily dominant request-complete pattern under a latency dashboard is exactly such an entry. The regulator exists for the heartbeat and the retry loop, which flood with nothing watching them. The engine also publishes every pattern's pre-move volume as a metric series and Datadog is among the backends, so the figure stays on a dashboard while the raw stream is split.

Exemptions, caps and levers live in committed files: an exemption is keyed by the pattern ID, a cap and a lever by the container, and every entry carries a reason and an expiry date, so a guard does not rot under the reformats that break a match string. Volume is tallied per pattern before and after regulation, which is the record a move is defended with later.

The log10x MCP server installs into an AI client and boots in demo mode against the live demo environment the table above was read from, though nothing there needs an account.

The sidecar has a cost of its own. Per node it runs about 250 MiB and 150m CPU, at sub-millisecond per event. Memory is where it breaks first. On one run, after 20.8M log records were forwarded, the sidecar was OOMKilled at a 3Gi limit, while the collector beside it stayed healthy at 1Gi. Sizing the sidecar above the collector it rides with is the practical reading of that run.

The tiers, the prices and the enforcement stay the platform's own, and the decision comes from the sidecar outside them, which reads every line before the platform receives it.

Related: Search compact logs in Splunk using the 10x app, Cutting Elasticsearch log storage roughly in half without changing queries, Expanding compact logs in ClickHouse with plain SQL.