Why agents need stable pattern IDs for logs

An agent investigating an incident fails on logs for one reason: the patterns it reasons about, groups of similar lines, are rebuilt every search, so nothing learned in one step names anything in the next. 10x names lines after the logging statement that printed them: one pattern hash holding across nodes and deploys under a pinned library and engine version. Once names persist, an agent can cut the bill per pattern, follow cause and effect between services, and pull dropped events from S3. The post ends on a Prometheus bug that reported a trivial pattern as a phantom +24900% spike.

An agent investigating an incident works in steps: rank what changed, correlate the top suspect against everything else, fetch the raw lines behind the best candidate. Every step is a separate tool call: the agent picks one of the functions it was given, fills in the arguments, and reads the result. To chain two calls, the agent has to name what the first one found when it fills in the second.

The automatic pattern views log platforms ship have no such name. Datadog Log Patterns recomputes its clusters from a 10,000-log sample of whatever the current search matched, and publishes no pattern identifier at all; Splunk's Patterns tab resamples every time it is opened. A person reading a dashboard never notices. An agent passing a reference between calls dereferences a name that no longer exists, and the toolset collapses into dozens of separate ways to grep.

Frequency-based pattern clustering is fine for a dashboard and broken as the foundation an agent stands on. A dashboard reader wants the loudest thing. An agent following a causal chain needs names that stay the same from one call to the next.

A stable name for every logging statement

The 10x approach is to assign that name in the pipeline, before any backend sees the line. The 10x engine runs as a sidecar to the log forwarder and resolves every line, as it passes, to one identifier I'll call the pattern hash: the same string for every line sharing a format, unchanged across queries, restarts, and deploys on every node sharing the pinned symbol library and engine version, for the lines that library covers. cart_cartstore_ValkeyCartStore is a readable label for one such pattern; the hash itself is an opaque ID derived deterministically from the line's source-derived symbols.

The rest of the pipeline keys on the same hash. Per-pattern volume and cost time series carry it as a label, and the raw events in your own S3 archive are fetched by it. A name an agent gets from one query is still valid in the next, an hour later or a deploy later.

Cutting the bill per pattern is the clearest case

A per-pattern decision stays attached to the logging statement it was made about. An agent that finds one pattern dominating a container's volume can move it to a cheaper tier the platform sells, offload it to an S3 bucket the account owns, or sample it under a cap that counts per pattern, and the decision still binds after the next deploy. A decision written against a recomputed cluster has nothing to bind to by the next search.

A capable model can already do the pattern half of this from a pasted sample, and an agent with shell access to the stack can go further: regex filters at the forwarder, byte counters into Prometheus, an S3 output for what it sheds. The part no agent can retrofit is the past: everything it wires up starts recording the day it is wired, and the per-pattern history and the shed-event archive answer questions about last week only because the pipeline was recording before anyone asked.

Naming has the same trap one level down. An agent can hash normalized lines and put a name on every one, but text-derived names stop holding: two logging statements that print the same text share one name, a statement whose text varies takes several, and a format change renames the pattern mid-history. The engine derives the name from the logging statement itself, so it stays one hash per statement through all of that, and that same hash is the meter's label, the filter's match key, and the archive's index, three artifacts a hand-built loop keeps aligned by hand.

Investigation is the second use of the same property. A causal chain is a list of names, which pattern moved first and which followed, and following it means handing a name from one tool call to the next.

How the MCP server runs the cost loop

Our MCP server puts those names in an agent's hands. MCP is the standard interface for giving a model callable tools; this server runs on the agent's side and connects to the pipeline, reading the per-pattern series and the S3 archive and turning decisions into engine config. Its tools compose because the names compose: no tool can invent or rename a pattern, and nothing re-clusters between calls.

The server is at github.com/log-10x/log10x-mcp, Apache 2.0. The Dev CLI tier tools, log10x_resolve_batch and log10x_dependency_check, work with no Log10x backend; the investigation tools need the paid Reporter and Receiver pipeline behind them.

log10x_top_patterns ranks patterns by byte volume with a cost overlay, from the per-pattern series the pipeline meters, and its delta column against last week answers whether a deploy added cost. log10x_pattern_mitigate takes one hash and returns which mitigation paths are reachable in the environment, dropping at the forwarder, muting or compacting at the engine, each marked enabled or disabled by what is deployed there. The chosen action becomes engine config for that one hash, the sidecar enforces it on every line in flight, and a rerun of top_patterns over the dropped cohort measures the savings after the fact.

What log10x_investigate does

You hand investigate an anchor and it works out what moved around it. The anchor is a string: a raw log line, a pattern hash, a service name, or the token environment for "investigate everything at once."

It classifies the anchor's time-series shape as acute, drift, or flat. Flat terminates with an honest empty result; acute branches into the correlation engine.

The correlation engine queries Prometheus for every other pattern in scope, shifting the comparison window by a few offsets. If a candidate's curve lines up with the anchor's shifted back sixty seconds, that candidate moved first. The offsets are in src/lib/correlate.ts:

const LAG_OFFSETS_SECONDS = [30, 60, 120, 180, 300];

Patterns that peaked before the anchor get linked into a chain ordered by lead time. Each link carries a confidence sub-score, all visible to the model:

// src/lib/correlate.ts
export interface ChainLink {
  mover: CoMover;
  /** Stat sub-score (0-1) — magnitude above noise floor. */
  stat: number;
  /** Lag tightness (0-1) — sharpness of the peak across offsets. */
  lag: number;
  /** Chain coherence (0-1) — how well this link fits the chain vs star pattern. */
  chain: number;
  /** Final per-link confidence = stat * lag * chain. */
  confidence: number;
}

Chain coherence rewards links that form a line of cause and effect and penalizes a hub where everything points at one node by coincidence. investigate emits this decomposition rather than let the model grade its own confidence, which rates high on everything with no relationship to the data.

One tool earns its place by refusing

log10x_discover_join finds the field that lines up a log pattern with its APM service. It scores how much the label values on each side overlap, using Jaccard similarity, and treats two labels as the same dimension when that overlap clears 0.7, relaxing to 0.4 for pairs whose names already alias the same dimension. Below the floor, it returns a structured no_join_available refusal instead of a temporal-only ranking. A correlation with no structural evidence is a coincidence.

A divisor trap in the Prometheus ranking

The rate-change ranking in investigate is a Prometheus topk query that divides the current rate by the baseline rate. In Prometheus, the baseline guard > noise_floor is a filter, not a true/false test: clear the floor and your actual value passes through as the divisor.

A guard set to the per-series noise floor itself, 0.001 events per second, is low enough that a baseline of 0.002, about seven events across the whole hour, clears it and passes into the denominator at face value. Divide a current rate of 0.5 by that and the ratio is 250x, which comes back as +24900% off an absolute rate that is trivial. Those phantom movers float to the top of the ranker, quoted with full confidence.

The fix is asymmetric on purpose

The fix guards the baseline side with a floor of ten times the per-series noise floor. The base floor is 0.001 events per second, so the guard is 0.01: thirty-six events an hour, or you don't qualify as a divisor.

The argument is in what we did not guard. The current side stays open, because a spike from a quiet baseline up to a moderate rate is a real incident. A crashlooping pod has a low absolute rate but is exactly what investigate should flag: the accounting service's Kerberos dlopen failure, a restart loop averaging 8 events an hour.

The server ships a NUMBERS DISCIPLINE block the model reads every session: don't recompute a percentage the tool already emits, don't invent peak values when the tool returns window averages, treat honest empty returns as a feature.

Tools tier with the infrastructure

A useful log assistant shouldn't be conditional on a vendor relationship, so the tools that need no Log10x deployment keep working with none. The Dev CLI tier needs no pipeline infrastructure. Paste lines from a Slack incident into log10x_resolve_batch for per-pattern frequency, severity, and variable concentration. Before deleting a log.info() call, log10x_dependency_check returns a script that scans your SIEM, dashboards, and alerts for anything still referencing it.

The paid tiers add tools only when the infrastructure can answer them. Reporter, the read-only fluent-bit DaemonSet, emits per-pattern cost and volume time series. Receiver is the sidecar that filters, samples, and compacts events in flight, and it sees the ones dropped before the SIEM gets them. Together they add top_patterns, pattern_trend, event_lookup, savings, and investigate itself. Retriever adds log10x_retriever_query, forensic retrieval of the events the Receiver held back, read from your own S3 bucket through pre-computed Bloom filters.

Install the server with npx -y log10x-mcp. The correlation engine is in src/lib/correlate.ts, the orchestration in src/tools/investigate.ts, the instructions block in src/index.ts. Wire it against your own pipeline and follow a chain from top_patterns through investigate into retriever_query.

Related: why Drain pattern IDs drift, what else is keyed to those patterns, and where log structure is defined. For the same data inside a SIEM, see the Elasticsearch, Splunk, and ClickHouse posts.