Searching offloaded S3 logs without an index server

Logs offloaded to an S3 bucket get searched like any other logs. Answering a query there means reading objects, and the usual way around that is a search service beside the bucket. The design in this post keeps the index inside the bucket. A summary of what each chunk holds sits in the name of an empty object. A query prunes by listing those names before reading any bytes.

When logs are offloaded to S3, the expensive part of a search is reading the objects, so the first job of any query targeting a bucket is deciding which objects not to read. Bloom filters are the standard tool for that: a small fixed-size summary can rule a chunk of data out for certain. Filters in this design are ordinary ones.

Each filter is serialized into the key of its own S3 object, as the last segment of the path, and the object itself is zero bytes. Pruning therefore reads only key names, so a query issues no GETs and stands up no index server beside a bucket that already covers durability and access. One paginated LIST streams a time window's filters back, up to a thousand per page.

Key length is the catch: an S3 key holds 1,024 bytes at most. Two things follow: few filters per time window, and each one small enough to fit in a key. A filter fed every word of every message would overflow the cap, though splitting it across more keys explodes the number of keys a query has to list.

Shrinking happens at ingest. Every log line is printed by a format string somewhere in code, and from one line to the next only the values change. 10x recognizes the format string behind each event and rewrites the event as a stable template hash, a short identifier for the line's constant skeleton, plus the values that filled it:

2025-10-02 00:17:22 +0000 [info]: #0 starting fluentd worker pid=18 ppid=6 worker=0

ships as

~R}>PZj;Jdp,1759364242000,0000,fluentd,18,6,0

That encoded form is what a plain SELECT expands back to the original text on ClickHouse. A filter takes in the hash plus the few values that varied, and the constant words never reach it.

Those events land in an S3 bucket in the account that owns the logs, which then answers queries by message identity with no log analyzer anywhere in the retrieval path.

Writing a filter into a key

Log events reach S3 through the forwarder that ships them, and the index writer runs beside that forwarder, processing the same events as they are offloaded.

Elements are gathered per time bucket and per byte range of the object being written. Those elements are the template hash of each event and each variable value as it appeared. Enrichment values the pipeline attached alongside the event (a service name, a pod) go in too, split into tokens. That whole set becomes one Bloom filter, built with the orestes-bloomfilter library.

A Bloom filter is a fixed-size bit array that answers a membership question: definitely absent, or possibly present. Elements set bits on the way in, and a lookup checks the same bits. Accuracy costs bits, though: a lower false-positive rate needs a bigger array, which matters once the filter has to fit in a key.

AWS caps an S3 object key at 1,024 bytes of UTF-8, and the filter has to fit inside that.

Serializing turns the filter into a printable string, which the index writer joins onto a path that already carries the routing facts a query needs. Those are an app prefix, the time bucket, a short hash naming the offloaded object, and the index of the byte range this filter covers. The filter string becomes the final path segment, and the writer PUTs the whole thing with a null stream and a content length of zero.

Fitting under the cap is a negotiation with the false-positive rate. Encoding runs at the configured false-positive target, 1% by default, and the builder measures the whole key, path plus filter, against the 1,024 bytes. At that default, though, there is no accuracy left to trade.

1% is both the rate the builder starts from and the loosest it will accept, so the first key that measures over the cap splits the element set in half. Each half is encoded and measured the same way until the pieces fit. Sibling filters for one byte range come from those splits, though no element is dropped to fit.

Message text is left out of the filters. Human-readable skeletons live once each in a template dictionary nearby, deduplicated by template hash and folded into merged snapshots. That dictionary holds the text once per message type instead of once per filter.

Resolving a hash back to its skeleton means listing the template dictionary and reading every unmerged file plus the newest merged one, because there is no lookup by hash. The writer folds them together on a timer, one minute by default, deduplicating on the template hash and deleting what it replaced.

Why the filters stay small

Templates also shrink the Bloom filters that route queries across S3 buckets, which is the claim the post on class knowledge left open. A keyword index has to answer for every word of every message, so the constant words dominate it: starting, fluentd, worker, repeated across millions of lines and all of it stored so text search can find it. Here those words never reach a filter, though.

At ingest the engine collapsed them into the template hash, a single element, leaving the handful of values that varied. A smaller element set needs fewer bits at the same false-positive rate. A 1,024-byte key has room for what is left, once its own routing path is accounted for. A filter built over every word would have needed a body of its own instead.

That only works because the hash holds still, and why it holds is its own post. Each hash is computed from the format string's fixed words and the punctuation around them, never from runtime values. Any host and any deploy that shares the pinned symbol library and engine version produces the same hash for the same message type, though only for the lines that library covers.

An index keyed on a grouping that drifts would strand every filter written before the drift, though the key here is the message's format string.

How a query prunes

Querying opens with ListObjectsV2 against the index prefix, a start-after cursor dropping the scan at the query window's first time bucket. Keys come back in lexicographic order, and the time bucket at the front of the variable path makes lexicographic mean chronological.

Each key parses back into routing facts plus a filter. The writer serialized the bit array into the key, and the query path rebuilds the same filter from the key string at query time.

Two tests run against that in-memory filter. One asks whether the filter could contain the queried template hash. A value clause evaluates against the same already-decoded filter, one more membership lookup, so it costs memory reads on data the LIST already delivered and adds zero S3 requests.

A query with no hash works only through the reserved text field. Its terms are matched against the template dictionary to resolve hashes, and tokenized against the same filter. A blank search is refused outright, so there is no unscoped scan.

A filter that passes both tests names its target: which offloaded object, and which byte range within it. Each such object then costs a small read of its byte-range index, an object that does have a body, to turn a matched filter into byte offsets.

Matched ranges against the same object collapse into a single ranged GET, spanning the start of the earliest range to the end of the last. Reading consumes the wanted ranges and skips the gaps in-stream. On those fetched bytes the query's predicate runs once more, exactly this time.

Correctness comes from that re-check. A Bloom filter never denies an element it holds, so a single filter yields false positives but never a false negative. A query for one message type misses no range that holds it, because that hash is stored whole in a single filter. An exact match on fetched bytes leaves every returned event a true match, and a false positive costs a wasted fetch rather than a wrong result.

Pruning bills out as the LIST itself, a page per thousand filters in sequence, so the width of the query window sets both the wait and the request count. A byte range full of unique values, request IDs and the like, splits into more sibling filters, and the LIST pays for each of them as another key.

One measured run indexed a 9 MB object in about six minutes. The filter it produced landed in the 150-to-199-byte band of the 1,024-byte key limit, holding fifty to ninety-nine elements.

Choosing keys over a service

The alternative was an index service: an OpenSearch cluster over the bucket, or a small database of postings per prefix, something with an endpoint. Each of those is a second stateful system, carrying its own durability, replication and access control, next to a storage service that already provides all three. I went with keys instead.

A filter stored in a key inherits every guarantee the bucket already has, and its read API is LIST, which any process holding the bucket owner's credentials can call. Between queries this index consumes no compute anywhere, though a query still pays for the LIST.

Cribl Search queries object storage by standing up search compute against the bucket and reading the objects' contents.

This design needs neither piece. Pruning is a LIST issued with the bucket owner's credentials, and retrieval is by stable message identity. Identity means a query for a message type finds every instance of it, whether or not anyone remembers the exact words the developer chose.

Choosing a Bloom filter over an inverted index came down to cost. A Bloom filter holds to a fixed budget of bits, and its one error mode, the false positive, is caught by the ranged-GET re-check. An inverted index gives exact term-to-location answers, though it pays for them in size and in merge work, both of which need a server to run on.

Retrieval keys here are computed from the message's format string, never learned from traffic. One message type resolves to the same key on every node and after every restart, for the lines the pinned library covers, and re-clustering does not move it. Logs offloaded six months ago answer to the identifier a query uses today, as long as the library and engine version that wrote them are still the pinned ones.

A filter stored in an object body would cost one GET per filter before a query learned anything. Stored in the key, filters arrive with the LIST response, up to a thousand at a time.

Related: what else attaches to a template, how far runtime cluster IDs drift, and the same hash expanded back to text in plain SQL on ClickHouse.