Sampling logs per message type with a severity floor

One message printed in a loop can dominate what a container sends to the log platform. The usual answer is blind sampling, which drops a rare error at the same rate as the flood. This post covers a different control: 10x groups log lines by message shape and trims only a shape that crosses its byte budget. Error events keep a floor rate.

Log volume is billed by the GB, and repetition drives it: a heartbeat or a retry printed in a loop can dominate a container's output. The standard control is to sample, keeping 1 in N events and dropping the rest. Because that rate is uniform, a message repeated hundreds of times and a single critical error are dropped at the same rate.

A critical error is the whole reason to keep the logs, and severity is the field that marks one. A sampler can be set to keep errors, and a rule can spare a named message, though both need the problem named ahead of time. Neither catches a pattern that floods for the first time overnight. In 10x, when volume has to go, the trimming lands on the flooding pattern, and the rest of the container passes untouched.

A pattern is a message shape. Most log lines are printed by a format string, so the same text repeats with only the values changing:

GET /api/v1/orders/8313 200 12ms
GET /api/v1/orders/4098 200 9ms

Both lines are one pattern. A stable pattern ID is stamped on each event by the paid 10x engine, which runs as a sidecar to the log forwarder. Two lines that differ only in their values carry the same ID.

Two volume controls come before any sampling, and both keep every event. One is compact, a lossless volume reduction on the forwarder that is expanded at search time, and the other is offload to object storage that still answers queries in place. Regulation sits under both, a last resort an operator opts into by setting a byte budget. An event the regulator drops is not parked anywhere, which is why it comes last.

The per-pattern byte cap

Regulation runs on the forwarder, before the volume reaches the log platform that meters it, and bytes are counted per pattern per container over a rolling window of 4 minutes by default. A pattern is left alone until its bytes in that window cross a cap the operator set, per container in a committed CSV or fleet-wide in one option. With no cap set, the regulator does nothing and keeps no counters, though the protection list below still applies.

The fleet-wide option is absoluteCap, one byte cap applied to every container, and the shipped value is 0, which means off. The per-container file carries two columns under the header container,cap:

container,cap
payment-service,4194304:1735689600:annual budget protection PAY-101

A cap can carry an expiry epoch and a reason after the byte figure, as in that row. A cap of 0 exempts the container it names.

At or below the cap, every event is kept. Above it, each further event of the over-cap pattern is kept at a rate set by its severity under the floors below, and every other pattern in the container passes through untouched.

Because the window is recent, a pattern that spikes during a deploy and then goes quiet stops being trimmed on its own. The first 5 events of every pattern are also kept each window, so even a heavily trimmed pattern leaves a sample to inspect.

Choosing which pattern to trim is all the cap does, so the kept volume can exceed the cap, though the loss stays on that one pattern. A pattern holding less than 5% of its container's recent volume is left alone even over the cap.

A container additionally runs unregulated for its first 5 minutes under a regulator instance. Counts start empty, and the small-pattern guard reads a ratio of them, which is noise without a few minutes of accumulation. Trimming a crash loop falls to the protection list instead, where a mute entry applies from the first event. Warm-up is per regulator instance, though, so a container that restarts faster than 5 minutes is never capped.

Severity floors

A pattern over its cap still keeps its Error and Fatal events at a 50% rate, Warn at 30%, and Info at 10%. The rate is a per-event keep probability, though, so a small window can come out slightly above or below it. A pattern whose events carry no severity keeps them at the Info rate, 10%.

Severity is information the line itself declared, and the floors read it parsed from the original line, whatever field it used.

Bytes are all the cap counts. No cap arithmetic pushes retention below a floor, so a byte rule cannot take a pattern's Error events to zero.

Floor rates are operator-settable, not only the caps. severityFloors holds the per-level map, and minRetentionThreshold covers every level the map leaves out, shipping at 0.1. Shipped floors are 0.5 for ERROR, CRITICAL and FATAL, 0.3 for WARN, and 0.1 for INFO, DEBUG and TRACE.

Floors are per event, so an anomaly inside the flooding pattern still has a way through. A line of the over-cap pattern that parses as Error keeps at its 50% rate, and the first 5 of every window survive regardless.

One case is left uncovered: the anomaly that changes only a value, carrying no severity signal, which a volume rule cannot see. Regulation comes last for that reason, after the controls that keep everything.

Where the cap leaks

The cap is a trigger rather than a bound. The floor check runs after the cap check, so a pattern flooding far above its cap keeps passing events at its floor probability. Half of its Error events and a tenth of its Info events keep going through, however far over the cap the pattern runs. Worst case for a container is the cap plus the floor share of everything above it. The shipped documentation says as much. An operator who reads the cap as the volume the platform will bill has the wrong model, and sizing has to allow for the floor share of a flood.

What the run kept

A behavior test pins this down. The run uses the shipped release with a 1 KB cap per pattern, and feeds one container, past its warm-up, three patterns. One is a flooding heartbeat with no severity, 201 events of about 130 bytes each, and the other two are 5 Error events and 20 GET request events, so that only the heartbeat's bytes cross the cap and only the heartbeat is trimmed. The regulator keeps 25 heartbeats, all 5 errors, and all 20 GETs.

Where the 25 comes from: the first 8 heartbeats passed whole, about the count that fits in 1 KB of 130-byte lines. Each one past the cap is kept with 10% probability, the no-severity floor, which on this run kept 17 of the remaining 193.

A blind sampler shedding the same volume keeps 50 of 226 events at one uniform rate. The regulator reaches the same total by taking all of its trim from the one pattern that was flooding. No rule named the heartbeat in advance: the budget was committed once, and the cap catches whichever pattern crosses it. That uniform rate, applied to 5 errors, keeps about 1 of them.

The protection list

An operator can name a pattern in a committed CSV, either exempting it from regulation or muting it fully, on an entry that carries an expiry and turns into a no-op once that passes. Gitops pulls the file, so every change arrives as a diff and a review. For a pattern the file names, the entry settles what happens ahead of the cap.

Severity floors apply to a full mute as well, and no entry in the file can lower one. A muted pattern still keeps events at the floor rate for its severity, 50% for Error and Fatal.

Keying on the pattern ID

Every rule above is a lookup against the pattern ID: the cap accumulates bytes per ID, the floors count retention per ID and severity, the protection list is keyed by ID, and the per-window sample is kept per ID. All of it needs the same message shape to resolve to the same ID from one window to the next, including across a restart.

Format variation inside a line does not fragment that ID. A pattern is a chosen subset of the line: the run of tokens the engine's coverage ranking traces back to one place in the code. Variable material is left out, so format variants of the same statement collapse to the same ID.

A clustering pass that reshuffles its groups as new lines arrive cannot hold a commitment. Per-pattern accounting resets whenever the groups move, and a mute merged against Monday's cluster points at a different grouping by Wednesday. Such a pass can still trim a flood, though, since within a single window its groups hold together well enough.

Editing a format string re-keys the pattern ID too, but that change is discrete and shows up in a code diff, while a reshuffle has no diff to review. Blind sampling has no per-pattern identity at all, so there is nothing for a cap, a floor, or a mute to attach to.

One identity serves both the regulator and the cost report. A pattern a report names as a container's top spender is the same key the cap trims, so a cost finding maps to a regulation target with no translation. The MCP tools an AI assistant uses to query these logs carry the same ID from one call to the next.

The regulator stays idle until a single pattern exceeds the byte budget its operator committed, and the trim then falls on that pattern. Defaults, floor rates and file formats are documented at doc.log10x.com.