Search compact logs in Splunk using the 10x app

The 10x app cuts the volume Splunk indexes and bills for, and keeps logs fully searchable. On the public otel-sample.log, it compacts 215 MB of raw lines to 78 MB counting the shared template dictionary, a lossless 63.7% reduction. Log lines from the same code path repeat the same fixed text and differ only in their values, so the app stores that text once and each event as just its values. At search time the 10x app expands each event back to its original line, and existing dashboards and saved searches return the same results. The whole app is open source under Apache 2.0: twelve lines of jQuery in the browser, a Splunk REST endpoint, and a pure-SPL macro.

Cutting Splunk volume usually costs information: drop noisy lines, sample, or strip fields. Most of that volume, though, is repetition. A single log.info("User {} logged in from {}", userId, ip) call site emits the same line a million times, and only the timestamp, the user, and the IP change. Write each event to the index as a hash of the shared template plus those three values, and it costs a fraction of the bytes yet expands back to the original line. The catch: a dashboard built against the original lines finds none of its search terms in the encoded events. We built 10x for Splunk to expand events back at search time, so existing dashboards and saved searches keep returning what they always did. The browser side is twelve lines of jQuery; the rest is a Splunk REST endpoint and the tenx-inflate macro.

The approach rests on one property: the hash that identifies a template must be stable, the same call site yielding the same hash every time. Templates inferred from a sliding window of live data drift as it moves, a poor storage key. We derive the vocabulary from the format strings and class names in the application's repos, so a compile pass yields a deterministic library, and the hash holds on every node that shares that pinned library and engine version. Where that library comes from is its own post; here I take the hash as stable across queries, deploys, and nodes sharing the library.

The reduction is lossless and specific to destinations that store the encoded form; and the Splunk app in this post is Apache 2.0 and inspectable end to end, while the Receiver that produces the encoded events is the commercial piece. On the public otel-sample.log, 215 MB of raw lines compact to 78 MB, counting the encoded events plus the template dictionary they expand through: a measured 63.7%. Round-tripping that sample returned all 197,430 lines byte-identical; the boundary sits at the file read, where bytes that are not valid UTF-8 are substituted before the engine tokenizes, the same substitution any UTF-8 log reader makes. Across the full measured benchmark the per-dataset spread runs -4.6% to 96.1% by content, so the number is a property of the stream, not a promise.

Now, search time. A panel that yesterday showed 2026-04-14 10:23:01 INFO admin logged in from 192.168.1.1 shows ~x7Kp2m,1776162181000,admin,192.168.1.1. Every SPL query needs editing, yet we had to expand events back without touching a saved search or dashboard.

Three places to intercept; we picked the hook

A custom search command is Splunk's official extension: add | tenxsearch to every query. Easy to build, but every saved search, panel, and alert needs editing: a deployment wall.

Expanding at index time sidesteps that. The forwarder expands events before indexing, so everything downstream works. But it pays for the full raw stream again: on an ingest-based license, Splunk meters the raw volume entering the indexing pipeline, the very bill 10x was meant to remove.

Hooking the browser's search submission is where we landed. It covers interactive search, the classic dashboards and saved-search dialogs that run in the browser; server-side searches never touch the browser, so they fall outside it. We point the URL the dashboard POSTs to at our REST endpoint, which rewrites the SPL, submits it, and returns a standard sid.

The interception is twelve lines

Splunk auto-loads dashboard.js from an app's appserver/static/ directory. Ours is 37 lines: it waits for the page, then calls TenxSearchHook.execute(true). The hook lives in tenx_search_hook.js, a 49-line module; its twelve interception lines:

// tenx-for-splunk/appserver/static/javascript/search/tenx_search_hook.js
$.ajaxSetup({
    beforeSend: function (xhr, settings) {
        if ((settings.type == "POST") &&
            (settings.url.endsWith("/search/jobs"))) {

            var baseUrl = settings.url.substring(
                0, settings.url.length - "/search/jobs".length);
            settings.url = baseUrl + "/tenx-search";
        }
    }
});

The handler creates the job through Splunk's own /search/jobs endpoint and returns the same {"sid": ...} payload that endpoint produces, so the browser gets a real Splunk job ID. The rewrite fires only on the POST that opens a job, so the status polls that follow, GETs against /search/jobs/<sid>, reach Splunk unmodified.

The remaining lines handle a startup race: some dashboards fire searches before the hook loads. The execute(true) call walks the search managers Splunk already instantiated, via SplunkJS's component registry, an internal API rather than part of the documented surface, plus startSearch({refresh: true}), and re-issues those through the intercepted path. The classic framework puts dashboard search submission through jQuery, which is what leaves a single place to intercept.

What /tenx-search does with the SPL

The REST endpoint is registered in restmap.conf as a persistent Python 3 handler, which hands the search to TenxSearchBuilder, resolved in four steps.

Parse the SPL. The builder calls Splunk's /services/search/parser endpoint with parse_only=true, then modifies only the leading search command. Everything after the first pipe passes through.

Check for encoded data. It inspects props.conf for the REPORT-tenx = tenx-hash-vars-extraction extraction that splits the encoded line into hash and variable slots. Sourcetypes lacking it use native search.

Find the matching templates. The search words are joined with OR and run against tenx_dml_pure; matches yield their hashes.

Build the combined search. In admin logged in, admin is data in the encoded event; logged in is template text, reachable only by hash. One clause catches each:

Original:
    search index=main sourcetype=tenx_encoded admin logged in

Resolved:
    search index=main sourcetype=tenx_encoded
        ((admin OR logged OR in) OR (tenx_hash IN (x7Kp2m,r9Qw3n)))
    | `tenx-inflate`
    | extract
    | search admin logged in

Here tenx_hash is the template hash under its on-the-wire name. The OR pulls in events matching either place; the trailing | search drops anything whose decoded _raw lacks the original terms.

On a shape it cannot model, the parser flips to COMPLEX, one of four states alongside SUCCESS, FAILURE, and PENDING, and submits the query unchanged. I would rather return raw encoded results than silently break a query.

Subsearches recurse: a bracketed subsearch is handed to a nested builder and resolved before the outer search is parsed. Fallback happens per command, so a parse error, a failed subsearch, or a failed template lookup each leave that command as the user wrote it. The exception is an unexpected error inside the builder, which replaces the search with a macro returning one event that names the failure.

The tenx-inflate macro reassembles events in pure SPL

tenx-inflate reconstructs events in pure SPL, with no Python. A regex in transforms.conf splits the event into hash, first variable, and the rest:

REGEX = ^~?(?<tenx_hash>[^,]+),(?<tenx_var_0>[^,]+)(?:,(?<tenx_vars>.*))?

The comma split is safe because a variable value never contains one: any comma in the source line is a delimiter, stored with the template, so the wire carries only the values between them.

The leading ~ marks an event as encoded. When the template has a timestamp, the first slot carries it, so the macro treats tenx_var_0 as the epoch.

makemv delim="," tenx_vars
| lookup tenx-dml-lookup _key AS tenx_hash
    OUTPUT part_0 AS tenx_part_0,
           pattern_parts AS tenx_log_parts,
           pattern_terminator AS tenx_log_term,
           timestamp_format AS tenx_ts_f
| eval tenx_ts_sec=if(tenx_var_0 > 10000000000000,
                       tenx_var_0 / 1000000000,
                       tenx_var_0 / 1000)
| eval _raw=if(isnull(tenx_log_term), _raw,
    if(tenx_ts_f == "",
        if(isnull(tenx_var_0), tenx_log_term,
            mvjoin(mvappend(tenx_part_0, tenx_var_0,
                mvzip(tenx_log_parts, tenx_vars, ""),
                tenx_log_term), "")),
        replace(
            mvjoin(mvappend(
                mvzip(tenx_log_parts, tenx_vars, ""),
                tenx_log_term), ""),
            "__TENX_TS__",
            strftime(tenx_ts_sec, tenx_ts_f))))
| fields - tenx_hash, tenx_log_parts, tenx_log_term,
           tenx_ts_f, tenx_part_0, tenx_var_0,
           tenx_vars, tenx_ts_sec

The lookup pulls the template's static text from the KV Store by _key; mvzip, mvappend, and mvjoin weld each segment and value back into _raw. For timestamped templates, replace swaps in a __TENX_TS__ placeholder, and a check against 10 trillion tells nanoseconds from milliseconds. A final | extract runs field extraction against the rebuilt _raw. Exact expansion needs one Receiver-side setting: varMaxRecurIndexes: 0 in the pipeline feeding Splunk. The app README covers the setting and its small trade-off.

Templates arrive via HEC and a two-minute cron

Templates enter Splunk via HEC as JSON on the tenx_dml_raw_json sourcetype. Each carries a templateHash and a template like User $ logged in from $, each $ a value slot. A saved search named "Consume KV" fires every two minutes, splitting each template on $ into the tenx_dml KV Store collection and writing a flat copy to tenx_dml_pure for matching. The cron is a latency gap: a new log format is not expandable until the next run picks it up.

Two expansion paths

The hook is the transparent path, and it serves whatever the browser submits: classic dashboards and the saved-search dialog in any app carrying dashboard.js. The file ships in the 10x app, and copying it into another app, Search & Reporting included, extends the same coverage there.

Everything else expands through the macro. Scheduled alerts run server-side, where no browser hook fires, so 10x compiles them once at save time: the authored search is rewritten into the same form the interactive path builds, the keyword-or-hash match piped through tenx-inflate, and stored as a native saved search the scheduler runs unchanged. REST calls from Python and SDK queries from external tools add tenx-inflate to the search themselves; code that reads events outside Splunk decodes them with the standalone Java or JavaScript decoder. Dashboard Studio, the newer React-based framework where Splunk is putting its new-visualization work, does not go through the classic jQuery stack either, so its searches take the macro path as well.

The full app is Apache 2.0 at github.com/log-10x/splunk-app; the setup guide covers install, HEC tokens, and forwarder config. Whether browser-level interception ages well as Dashboard Studio grows is the open question.

Related: why the template hash is stable, the same problem in Elasticsearch, solved one layer down inside Lucene, and the same idea on ClickHouse in plain SQL.