Extracting log format strings from source: ANTLR vs native parsers

A production codebase mixes several languages, and the log format strings 10x reads sit in all of them. Getting them out means one generic scanner covering every language, or a dedicated scanner per language calling the parser that language already ships. This post builds both, times them over the same Java codebase, and lands on the language's own parser. The generic scanner stays the fallback.

Production source is polyglot, and five-plus languages is normal. Extracting a symbol vocabulary from it means one scanner per language or one polyglot scanner. We built the polyglot one, then wrote three native scanners anyway, and this post is why.

The vocabulary is what those scanners produce: identifiers and literal strings pulled from code (class names, severity levels, the format strings inside log.info(...) calls), each tagged with its source location. The tag is what attribution runs on: if line A came from a method whose source contains the literal connection reset and line B did not, the tag attributes the line to the path that could have emitted it.

The case for reading the vocabulary from code instead of inferring it at ingest is made in the compiler post. This post is about the scanner choice. The scanner configs are MIT licensed in the open config repository. The engine that runs them is the paid product and is not in that repo.

The generic ANTLR scanner

ANTLR is a parser generator: feed it a grammar and it turns source files into a parse tree. We walk that tree and grab the text at node types we care about. The shipped configs are at config/pipelines/compile/scanners/antlr/, one per language, the Java one abridged here:

# config/pipelines/compile/scanners/antlr/java.yaml
antlr:
  lang: java
  parserClass: com.log10x.antlr.generated.java.Java9Parser
  rootRule: compilationUnit
  rule:
    - name: normalClassDeclaration
      context: class
      recursive: false
      capture: allSymbols
    - name: methodInvocation
      context: method_invoke
      recursive: true
      capture: literalsOnly

Each rule pairs a grammar rule name with a context and a capture mode. Context is the symbol's label (CLASS, ENUM, METHOD_INVOKE). Capture mode picks the tokens: literalsOnly takes quoted strings, allSymbols also takes identifiers. The recursive flag sets the depth: false grabs the class name, true takes every literal inside.

The Go config is identical in shape: a different parser class and rule names over the same taxonomy and walker. Adding a language means a YAML file and a grammar, no Java code.

Rules can also span multiple parse-tree nodes, and Python enums are the clean example: member names come only from classes extending Enum. A tag/ifTag pair in python.yaml gates the rule on a matching parent.

Why native scanners exist

Alongside the generic ANTLR scanner, three languages get a dedicated scanner that calls a parser built for that language, and the dedicated one wins unless disabled: Java has javaParser (JavaParser, an open-source library), Python has pythonAST (the built-in ast), Scala has scalameta.

We built the ANTLR scanner first, for Java, and it worked until we ran it against a production-scale codebase. ANTLR does work per file: lexer pass, parser pass, tree walk, scope tracking, tag resolution. That work adds up across a large repository.

Scanning the core of Guava, 611 Java files and 181,693 lines, on one laptop: the ANTLR scanner takes about seven minutes, and javaParser takes thirteen seconds on the same scan.

Two caveats scope that number.

The first is that it is not like for like. With the ANTLR rule set we ship, the two scanners do not emit the same thing: javaParser produced 2,328 named symbols on those files. ANTLR produced none, and about a quarter as many token groups. So ANTLR was slower and, as configured, saw less. An equal comparison would need a richer ANTLR rule set, and that rule set would be slower still.

The second is that we had to raise our own timeouts to let ANTLR finish. At the shipped defaults, one run took nine minutes and killed six scan subprocesses outright, all of them on Guava's largest collection files, and javaParser never did that.

So we wrote javaParser, which reads Java source with the JavaParser library and walks the AST that library returns. Nothing else about it is smarter: it labels a call site as logging the same way the ANTLR scanner does, by testing the invoked method name against the configured set of logging method names.

Use a language's own maintained parser where one exists, because scan time is the argument. ANTLR carries C and C++, C#, Go, JavaScript, TypeScript and Rust from one shared tree-walk routine plus a folder of YAML, and stays a fallback when a native build fails.

Ruby, Kotlin, PHP, Swift, Lua, Groovy and bash get quoted-string extraction only.

The configs sit alongside the C++ and Scala grammars at pipelines/compile/scanners/antlr. The other languages' ANTLR parsers ship pre-generated in the 10x engine, with docs at doc.log10x.com/compile/scanner/antlr. Related: where the vocabulary comes from, how a Drain pattern group changes, and stop extracting log fields per event.