Reading ClickHouse logs from the table and the bucket

Keeping logs in ClickHouse costs little. The part of the bill that grows with volume is the compute spent writing rows, and merging them afterwards. Most of those rows come from a handful of log statements that print far more than the rest of the code. Each of those statements gets a budget at the forwarder. Lines over budget go to an S3 bucket in the account, and one Merge table answers a query from both sides.

Why a ClickHouse log bill grows

Most of a ClickHouse log bill is the cost of writing rows, and only a little of it is the cost of keeping them. Every row that arrives has to be parsed, indexed, written to disk, and merged again each time parts combine, with the metadata tracked in Keeper. All of that is compute, and it is spent before the row reaches the disk where compression makes it cheap. ClickHouse makes the same point about managed ClickStack, where retention "effectively stops being a meaningful cost dimension" and the remaining variable "becomes compute".

The levers ClickHouse offers for a log table all act after that work is done, so none of them give any of it back. A TTL rule that moves older parts to cold storage runs in the background, by which time the parsing, indexing and merging are already paid for. Compressing the message column harder does little either. On the ClickStack schema the skip indexes and the attribute maps around the message hold most of the bytes, and the message itself is a minority of the row.

Where the rows come from

The rows are not spread evenly across the code. A Kubernetes controller logs every reconcile, an ingress writes a line per request, a Kafka broker records producer state on every snapshot. None of that is a mistake; it is what those components are supposed to do. The numbers in this post come from a test run over a public OpenTelemetry demo capture. In that run, twenty-one log statements produced two thirds of all the lines, and everything else in the code shared the remaining third.

For a statement like that, the choice available today is to keep it or drop it. Keeping it means paying the full write path for every one of those lines. Dropping it at the collector means the record is missing during the next incident, when someone goes looking for exactly that pattern. A rule there works by service or by severity. It has no way to tell one statement's flood from the lines beside it.

What 10x does instead

10x is a cost engine that runs beside the log forwarder, inside the account's own network. Events go to it from the forwarder and come back marked, so the shipping never changes hands and 10x only decides what a line is worth. Each statement gets a budget. Lines under budget go into ClickStack unchanged. Anything past the budget goes to an S3 bucket in the account.

For any of that to work, the same statement has to be recognisable from one day to the next. Most log tooling works this out by clustering similar lines together as a query runs. The same line can land in one group today and a different one tomorrow, as traffic shifts. A budget set against yesterday's group no longer applies to the statement it was written for. 10x separates the fixed wording in a line from the values that change, and builds the identity from that half alone. Two runs of the same statement a month apart hash to the same value, whatever changed in between.

A line that goes over budget carries that identity with it, along with a mark saying where it should go. Those lines are written to S3 as files, one set per service per day. A Merge table over the two sides answers a query from both at once. HyperDX can be pointed at it as a second source, with the plain ClickHouse table still the default, and a column on each row names which side it came from, which is also how a query asks for one side only.

Results from the test run

That same capture went through end to end. About a third of the records landed in the ClickHouse table and about two thirds in the bucket; the two sides add back up to the count that went in. Every message type that appeared on the wire turned up in storage, on one side or the other.

What a query over S3 costs

Each file in the bucket carries the service and the date in its name. ClickHouse can skip a whole file on the name alone. It never opens it. Filtering by time works differently: the timestamps sit inside the files rather than in their names, so a query bounded only by time has to open every file in the range to find out which of them hold anything relevant, and over a month that means all of them. Across a month of accumulated files, that is the difference between a query taking seconds and a fraction of one. A dashboard should filter on service or date alongside its time range.

File size matters more than it looks. On an early run we took the batch processor out of the collector, which seemed harmless, and the cold side went from 65 files to 1,180 for the same rows. Every one of those is a PUT going in and a GET coming back out.

Batch size is set in the collector config.

Counting lines on both sides

A chart showing how often a statement fires has to read both sides. Once part of a statement's output is going to S3, the ClickHouse table on its own holds about a seventh of those lines. A chart built on that table alone would show the drop as though the traffic had stopped. Reading through the Merge table gives the full count. Filtering on the side column brings the cost back to roughly what ClickHouse alone would charge.

Running the harness

The run behind these numbers is published at github.com/log-10x/benchmarks. One command starts ClickStack 2.38.0, engine 1.1.79, a stock OpenTelemetry collector and MinIO (pinned by digest, so the run repeats). The same command pushes the capture through and writes into the results folder the reconciliation, the query costs, the object list, and every count and millisecond this post rounds.

Related: how a log line gets a stable pattern ID, sampling logs per message type with a severity floor, and an S3 index built out of object keys.