Why an agent needs a stable pattern ID

Dor Levi

Dor Levi

— 6 min read
Share
An agent cutting a log bill works in steps days apart: rank the expensive messages, cap one, check a week later. A rule written against the text of a log line stops matching once a deploy rewords the line. This post walks the four MCP calls behind that work, each keyed to an ID 10x reads off the source statement.

Cutting a log bill means finding the messages that dominate the volume, capping or rerouting the worst of them, and checking a week later what came off. The work is repetitive and measurable. Agents are being pointed at the job already, and the fit is plain, since every step is a query or a config change.

The catch is the week in between. Whatever the agent capped, it capped by matching the text of a log line, though any deploy inside that week can reword the line and stop the match. The agent comes back to a count of zero, reports the saving, and cannot tell that reading apart from the message having gone away.

10x runs in-path with the log forwarder (Fluent Bit, the OpenTelemetry Collector, and others), so it sees every log line before the log platform does. The engine gives every line printed by the same statement in the source the same ID, then reports volume against those IDs. Edit the wording of a message and its text changes, but the line of code printing it does not, so the ID an agent wrote a rule against last week still means the same message this week.

What a deploy breaks

The OpenTelemetry Collector in this pipeline prints an exporter failure, carrying the exporter version and the source line number:

2026-07-27T16:10:03.063Z  error  opensearchexporter@v0.142.0/logger.go:36  Request failed.
  {"resource": {"service.name": "otelcol-contrib", "service.version": "0.142.0"},
   "otelcol.component.id": "opensearch",
   "path": "/_bulk",
   "duration": 0.003685783,
   "reason": "dial tcp: lookup opensearch on 172.20.0.10:53: no such host"}

Upgrading the exporter moves both, to v0.143.0 and line 41. I wrote a regular expression against the earlier form and fed it 20 of these lines, 10 from before the upgrade and 10 from after. The expression matched the 10 earlier ones and none of the later ones.

10x keys on the part a developer wrote, the class, the method and the message text, rather than on the values around it. That part comes out of a library compiled from source, binaries and container images, so a message has an ID before it is ever printed. I ran those same 20 lines through it three times, once on the earlier 10, once on the later 10, once on all of them, and every run returned one ID, with the version and line number recorded as values:

"input_line_count": 20,
"resolved_pattern_count": 1,
"template_hash": "2qiwE{k$rY",
"slots": [
  { "inferred_name": "slot_1",
    "top_values": [ {"value":"142","pct":0.5}, {"value":"143","pct":0.5} ] },
  { "inferred_name": "logger.go",
    "top_values": [ {"value":"36","pct":0.5}, {"value":"41","pct":0.5} ] }
]

Running that check needs no pipeline and no account, since `log10x_resolve_batch` runs the engine over lines pasted straight into it, and works on a single line too, which is where grouping tools cannot follow. Drain3, the library several log platforms build their pattern views on, works by comparing lines to each other, so from one line it can only return that line, timestamp included, matching nothing else until a second occurrence teaches it which parts vary.

The four calls

These are functions on an MCP server, which is how an AI assistant is handed things it can call. Each call takes an ID the previous one returned.

1. Rank. `log10x_top_patterns` returns the messages ordered by volume, each with its ID. Asked for the side that never reached the platform, it returned 161 message types totalling 88.8 GB, out of 182.2 GB in 24 hours, which puts close to half the volume outside any export pulled from the platform, because the platform never received it.

2. Attribute. log10x_services splits each service's bytes by what the engine did with them, so the agent can see where that held-back volume sits before it touches anything:

service                    passed  offloaded  compacted   dropped
product-reviews          11.25 GB          0          0  32.04 GB
cart                      4.98 GB   37.23 GB          0         0
payment                   6.79 GB   10.06 GB          0   6.64 GB
llm                       4.51 GB          0          0  13.95 GB
opentelemetry-collector   0.54 GB          0          0   9.33 GB

3. Act. `log10x_configure_engine` takes a service and a percentage and returns a plan. Given payment, which runs 705.3 GB a month, and a 40% cut, it came back with three message types worth capping, keyed to the IDs the first call returned. The plan ships as a pull request. Once merged, the engine picks it up on its next poll with the pipeline still running.

4. Verify. A week later, log10x_estimate_savings compares the week before against the week after and splits the difference four ways: what the caps removed, what drifted, what arrived as message types that did not exist before, and what got through anyway. On payment, the caps accounted for 46.53 GB.

Step four is where the chain holds or quietly fails, reading the same IDs the first call handed over, a week and any number of deploys later. Had those IDs come from log text, a deploy inside the window would change what the two halves of the subtraction were counting, and the drift the split exists to expose would disappear into the result.

Levers by platform

The same ID supports different actions, depending on what each platform can do about a message once one is singled out, so the agent asks which of them apply before it plans anything.

  • Datadog bills for the events it indexes. Under that meter a smaller event costs exactly what a large one does. Move the message to Flex Logs, the cheaper tier Datadog already sells, or route it out to a customer-owned S3 bucket.
  • CloudWatch bills for the bytes it ingests and hands back the text it was given. Shrinking saves nothing. Move the message to Infrequent Access, at half the ingest price, or to an S3 bucket the account owns, though the routing rule has to exist on the CloudWatch side before the cheaper rate applies.
  • Splunk takes either. Routing to S3 keeps the message out of the index entirely. Shrunk events stay searchable where the 10x app is installed, because the app expands them at search time.
  • Elasticsearch, self-run, takes either as well. The 10x plugin cuts what sits on disk by roughly half with existing queries unchanged. Managed Elasticsearch, though, cannot run that plugin, so S3 is the only route there.
  • ClickHouse takes the S3 route. A message over its cap goes to a bucket you own instead of the table, so it never costs insert or merge work, and a Merge table reads it back beside the hot rows, more slowly than the hot table alone.
  • Azure Monitor bills most to ingest into the Analytics plan. Move the message to the cheaper Basic or Auxiliary table plan, or offload it to an S3 bucket you own.
  • Coralogix prices each gigabyte by the priority class it lands in. Move the message from High to Medium, where it stays queryable with alerting and dashboarding, or offload it to a bucket the account owns. Nothing forks to reach the cheaper class, though, so the decision has to arrive as a field on the event for the policy to read.

I ran this on a pipeline that ships to CloudWatch, which is why the compacted column above reads zero on every row, and why the plan moved messages to a cheaper tier instead.

One wrong turn belongs on the record. Until version 1.29.22, the first call returned nothing on every correct install. The boot probe counted a metric no engine emits, so each deployment reported itself as too fresh to analyse, and the ranking call came back with no data. That call also caps limit at 50, defaults it to 10, and sets a truncated flag on the response.

Reading capped messages back

Capping a message means one of two things here, and the engine's record distinguishes them. Routing sent cart's 37.23 GB to a customer-owned S3 bucket, where it still exists, while the collector's 9.33 GB was discarded at the forwarder and is gone.

The routed side stays addressable by the same ID the cap was written against, which is what lets the agent go back for it. log10x_overflow_contents lists that bucket by message type and log10x_retriever_query reads the events behind one of them, through an index held in the S3 object keys.

Installing the tools and the engine

The MCP server is at github.com/log-10x/log10x-mcp, MIT licensed, and installs with npx -y log10x-mcp. The counts it reads come from the engine in the pipeline, which the install guide covers putting there. The local check needs neither, so the upgrade test above runs on pasted lines alone.

Read more