Extracting log format strings from source: ANTLR vs native parsers

10x, a log analyzer, matches log lines against a vocabulary scanned from source: identifiers and literal strings (class names, format strings inside log.info(...) calls), each tagged with its source location, so a line can be attributed to the code path that could have emitted it. The post compares two scanner designs: a generic ANTLR walker with a YAML config per language, or dedicated scanners calling the language's own parser. On Guava's core, 611 Java files, ANTLR took about seven minutes against JavaParser's thirteen seconds, 32x, and as configured it extracted less. A maintained native parser wins; ANTLR stays the fallback.

Production source is polyglot; five-plus languages is normal. To extract a symbol vocabulary from it, you write one scanner per language or one polyglot scanner. We built the polyglot one, then wrote three native scanners anyway. This post is why.

The vocabulary is what those scanners produce: identifiers and literal strings pulled from code (class names, severity levels, the format strings inside log.info(...) calls), each tagged with its source location. The tag is what attribution runs on: if line A came from a method whose source contains the literal connection reset and line B did not, the tag attributes the line to the path that could have emitted it. The case for reading the vocabulary from code instead of inferring it at ingest is made in the compiler post; this post is about the scanner choice. The scanner configs are Apache-2.0 in the open config repository; the engine that runs them is the paid product and is not in that repo.

One tree walker, one YAML per language

ANTLR is a parser generator: feed it a grammar and it turns source files into a parse tree. We walk that tree and grab the text at node types we care about. The shipped configs are at config/pipelines/compile/scanners/antlr/, one per language. The Java config, abridged:

# config/pipelines/compile/scanners/antlr/java.yaml
antlr:
  lang: java
  parserClass: com.log10x.antlr.generated.java.Java9Parser
  rootRule: compilationUnit
  rule:
    - name: normalClassDeclaration
      context: class
      recursive: false
      capture: allSymbols
    - name: methodInvocation
      context: method_invoke
      recursive: true
      capture: literalsOnly

Each rule pairs a grammar rule name with a context and a capture mode. Context is the symbol's label (CLASS, ENUM, METHOD_INVOKE). Capture mode picks the tokens: literalsOnly takes quoted strings, allSymbols also takes identifiers. The recursive flag sets the depth: false grabs the class name, true takes every literal inside.

The Go config is identical in shape: a different parser class and rule names over the same taxonomy and walker. Adding a language means a YAML file and a grammar, no Java code.

Rules can also span multiple parse-tree nodes. Python enums are the clean example: you want member names only from classes extending Enum. A tag/ifTag pair in python.yaml gates the rule on a matching parent.

Why three languages got a dedicated scanner

Alongside the generic ANTLR scanner, three languages get a dedicated scanner that calls a parser built for that language: Java has javaParser (JavaParser, an open-source library), Python has pythonAST (the built-in ast), Scala has scalameta. The dedicated one wins unless disabled.

We built the ANTLR scanner first, for Java; it worked until we ran it against a production-scale codebase. ANTLR does real work per file: lexer pass, parser pass, tree walk, scope tracking, tag resolution. That work adds up across a large repository.

Here is the measurement. Scanning the core of Guava, 611 Java files and 181,693 lines, on one laptop: the ANTLR scanner takes about seven minutes, and javaParser takes thirteen seconds. Call it 32x on the scan phase.

Two caveats scope that number.

The first is that it is not like for like. With the ANTLR rule set we ship, the two scanners do not emit the same thing: javaParser produced 2,328 named symbols on those files; ANTLR produced none, and about a quarter as many token groups. So ANTLR was slower and, as configured, saw less. An equal comparison would need a richer ANTLR rule set, and that rule set would be slower still.

The second is that we had to raise our own timeouts to let ANTLR finish. At the shipped defaults, one run took nine minutes and killed six scan subprocesses outright, all of them on Guava's largest collection files. javaParser never did that once.

So we wrote javaParser; it reads Java source with the JavaParser library and walks the AST that library returns. Nothing else about it is smarter: it labels a call site as logging the same way the ANTLR scanner does, by testing the invoked method name against the configured set of logging method names.

If a language has a maintained parser of its own, use it; scan time is the argument. ANTLR carries Go, C++, C#, and JavaScript from one shared tree-walk routine plus a folder of YAML, and stays a fallback when a native build fails.

TypeScript has no scanner yet, native or ANTLR.

The configs sit alongside the C++ and Scala grammars at pipelines/compile/scanners/antlr; the other languages' ANTLR parsers ship pre-generated in the 10x engine. Docs are at doc.log10x.com/compile/scanner/antlr. Related: where the vocabulary comes from, how a Drain pattern group changes, and stop extracting log fields per event.