Sampling logs per message type with a severity floor

A log message printed in a loop can dominate a container's volume, and the standard fix, blind 1-in-N sampling, drops a critical error at the same rate as the noise. 10x groups log lines by pattern, the message shape that stays constant while values change, and regulates volume per pattern on the forwarder: nothing is trimmed until a pattern exceeds the byte budget its operator set, and severity floors keep Error events at a 50% rate even then. Fed 201 heartbeats plus 5 errors and 20 GETs under a 1 KB cap, the regulator kept 25, 5, and 20.

Log volume is billed by the GB, and repetition drives it: a heartbeat or a retry printed in a loop can dominate a container's output. The standard control is to sample, keep 1 in N events and drop the rest. That rate is uniform, so a message repeated hundreds of times and a single critical error are dropped at the same rate.

A critical error is the whole reason to keep the logs, and what makes it matter is its severity, not its volume. A sampler can be set to keep errors, and a rule can spare a named message, but both need the problem named ahead of time. Neither catches a pattern that floods for the first time overnight. In 10x, when volume has to go, only the flooding pattern is trimmed; the rest of the container passes untouched.

A pattern is a message shape. Most log lines are printed by a format string, so the same text repeats with only the values changing:

GET /api/v1/orders/8313 200 12ms
GET /api/v1/orders/4098 200 9ms

Both lines are one pattern. The 10x engine, the paid product, runs as a sidecar to the log forwarder and stamps each event with a stable pattern ID; two lines that differ only in their values carry the same ID.

None of this makes sampling the default. The volume controls that come first keep every event: the forwarder can compact the stream, a lossless volume reduction expanded at search time, or offload it to object storage that still answers queries in place. The regulation below sits under both, a last resort an operator opts into by setting a byte budget, and it drops nothing until a single pattern exceeds that budget. An event the regulator drops is not parked anywhere, which is why it comes last.

A pattern is trimmed only after it exceeds its byte cap

The regulator runs on the forwarder, before the volume reaches the log platform that meters it, and counts each pattern's bytes per container over a rolling window, 4 minutes by default. A pattern is left alone until its bytes in the window cross a cap the operator set, per container in a committed CSV or fleet-wide in one option; with no cap set, the regulator does nothing and keeps no counters, though the protection list below still applies. At or below the cap, every event is kept. Above it, each further event of the over-cap pattern is kept at a rate set by its severity, the floors below, and every other pattern in the container passes through untouched.

The window is recent, so a pattern that spikes during a deploy and then goes quiet stops being trimmed on its own. The first 5 events of every pattern are also kept each window, so even a heavily trimmed pattern leaves a sample to inspect. A pattern holding less than 5% of its container's recent volume is left alone even over the cap; a small pattern inside a busy container is not a flood. The cap only chooses which pattern to trim, so the kept volume can exceed the cap; the loss stays on that one pattern.

A container additionally runs unregulated for its first 5 minutes under a regulator instance. The counts start empty, and the small-pattern guard reads a ratio of them; without a few minutes of accumulation that ratio is noise. The warm-up is per regulator instance, so a container that restarts faster than 5 minutes is never capped; trimming a crash loop is the protection list's job, and a mute entry applies from the first event.

Severity floors outrank the cap

A pattern over its cap still keeps its Error and Fatal events at a 50% rate, Warn at 30%, and Info at 10%; the rate is a per-event keep probability, so a small window can come out slightly above or below it. A pattern whose events carry no severity keeps them at the Info rate, 10%. The floors read the severity parsed from the original line, whatever field it used, and no cap arithmetic pushes retention below them.

The cap is a rule about volume. Severity is information the log itself declared, and a volume rule that can zero out Error events cannot be trusted during an incident.

The floors are per event, so an anomaly inside the flooding pattern is not automatic loss: a line of the over-cap pattern that parses as Error still keeps at its 50% rate, and the first 5 of every window survive regardless. The residual risk is the anomaly that changes only a value, with no severity signal; a volume rule cannot see it. That risk is why regulation is the last resort, used only after the controls that keep everything.

Fed 201 heartbeats, 5 errors, and 20 GETs, the regulator keeps 25, 5, and 20

The behavior test that pins this down runs the shipped release with a 1 KB cap per pattern and feeds one container, past its warm-up, three patterns: 201 events of a flooding heartbeat pattern with no severity, about 130 bytes each, 5 Error events, and 20 GET request events. The regulator keeps 25 heartbeats, all 5 errors, and all 20 GETs. The heartbeat is the only pattern whose bytes cross the cap, so it is the only one trimmed; the errors and the GETs pass through whole. The 25 follows from the rules above: the first 8 heartbeats passed whole, about the count that fits in 1 KB of 130-byte lines, and each one past the cap is kept with 10% probability, the no-severity floor; this run kept 17 of the remaining 193.

A blind sampler shedding the same volume keeps 50 of 226 events at one uniform rate, and 50 of 226 applied to 5 errors keeps about 1 of them. The regulator reaches the same total by taking all of its trim from the one pattern that was flooding. No rule named the heartbeat in advance: the budget was committed once, and the cap catches whichever pattern crosses it.

The protection list overrides the cap, and a full mute still keeps Error and Fatal

An operator can name a pattern in a committed CSV and either exempt it from regulation or mute it fully; each entry carries an expiry, after which it becomes a no-op. The file is pulled via gitops, so every change is a diff and a review. A listed pattern is decided by its entry: the human declaration wins over the cap.

The severity floors apply to a full mute as well: a muted pattern still keeps events at the floor rate for their severity, half for Error and Fatal. The floors are deliberately not overridable: a mute aimed at volume should not be able to silence a Fatal.

A drifting cluster ID cannot hold a mute

Every rule above is a lookup against the pattern ID. The cap accumulates bytes per ID, the floors count retention per ID and severity, the protection list is keyed by ID, and the per-window sample is kept per ID. All of it needs the same message shape to resolve to the same ID from one window to the next, including across a restart.

The ID also does not fragment when a line's format varies. The pattern is a chosen subset of the line, the run of tokens the engine's coverage ranking traces back to one place in the code, and the variable rest is left out, so format variants of the same statement collapse to the same ID.

A clustering pass that reshuffles its groups as new lines arrive can still trim a flood; within a single window its groups hold together well enough. What it cannot do is hold a commitment: a mute merged against Monday's cluster points at a different grouping by Wednesday, and per-pattern accounting resets whenever the groups reshuffle. The pattern ID re-keys too, when a developer edits its format string, but that change is discrete and shows up in a code diff; a reshuffle has no diff to review. Blind sampling has no per-pattern identity at all, so there is nothing for a cap, a floor, or a mute to attach to.

The pattern ID the regulator keys on is the same identity the cost report attributes spend to. The pattern a report names as a container's top spender is the same key the cap trims, so a cost finding maps to a regulation target with no translation. The MCP tools an AI assistant uses to query these logs carry the same ID from one call to the next.

The defaults, the floors, and the file formats are documented at doc.log10x.com. The regulator does nothing until a single pattern exceeds the byte budget its operator committed. When one does, the loss is confined to the pattern that caused it.